Discussion:
Init : someone could comment this ?
(too old to reply)
Tanguy Eric
2008-01-05 16:20:54 UTC
Permalink
I just find this
http://www.pardus.org.tr/eng/projeler/comar/SpeedingUpLinuxWithPardus.html maybe it's known but someone could comment this because it seems to gain init time.

Eric
Casey Dahlin
2008-01-06 00:16:03 UTC
Permalink
Post by Tanguy Eric
I just find this
http://www.pardus.org.tr/eng/projeler/comar/SpeedingUpLinuxWithPardus.html maybe it's known but someone could comment this because it seems to gain init time.
Eric
There is work being done in this area for Fedora as well. I've been
working on a parallel booting system for us that also will provide dbus
notifications for the starting of various services. (Hopefully I'll be
leading a hackfest at FUDCon in a few days to get a few more eyes and
hands on the code).

We have had discussions on this in the past and had similarly decided
not to replace the init daemon. We also decided that we wanted to retain
compatibility with conventional sysvinit scripts.

As for rewriting some of the scripts themselves in a non-bash language,
there may be an advantage. I don't know if I like python in particular
for this roll (maybe something like Haskell that has proven to be light
and quick as well as nicer to deal with) but python is very entrenched
in Fedora, and I believe RPM/YUM still depends on it, so there's a low
cost to using it. Bash does a lot by running other programs, and that is
a lot of extra IO.

Those are what I see to be the points in the article that are relevant
to Fedora. The other elements are interesting, but not directly related
to the general aspect of improving boot speed.

--CJD
Jonathan Underwood
2008-01-06 00:46:42 UTC
Permalink
Post by Casey Dahlin
There is work being done in this area for Fedora as well. I've been
working on a parallel booting system for us that also will provide dbus
notifications for the starting of various services. (Hopefully I'll be
leading a hackfest at FUDCon in a few days to get a few more eyes and
hands on the code).
Other than compatability with sysvinit scripts, why was upstart dismissed?
Casey Dahlin
2008-01-06 03:26:17 UTC
Permalink
Post by Jonathan Underwood
Post by Casey Dahlin
There is work being done in this area for Fedora as well. I've been
working on a parallel booting system for us that also will provide dbus
notifications for the starting of various services. (Hopefully I'll be
leading a hackfest at FUDCon in a few days to get a few more eyes and
hands on the code).
Other than compatability with sysvinit scripts, why was upstart dismissed?
Many options were looked at during the last iteration of this
discussion, and by my understanding prcsys was the one that won out. See
here:
http://fedoraproject.org/wiki/FCNewInit/RC?highlight=%28fcnewinit%29
There's a few issues with prcsys internally that make it very difficult
to add the features we want (dbus etc), as well as some not-so-trivial
implementation issues (use of pthreads in a circumstance under which
they were a very poor choice), so Harald Hoyer and myself decided that a
rewrite was in order, so I went ahead and started working on a new app
(which I have titled 'rrn') which is now near feature parity with
prcsys. My academic schedule has meant I haven't had enough of it done
to be worth showing to anyone until very recently, so I've not been
talking too much about it here, but now that its starting to come
together, I'd love to start the discussion up again (and of course you
can all tell me I'm crazy and to take my little program and go home :).

--CJD
Nicolas Mailhot
2008-01-06 10:50:18 UTC
Permalink
Post by Casey Dahlin
There's a few issues with prcsys internally that make it very difficult
to add the features we want (dbus etc), as well as some not-so-trivial
implementation issues (use of pthreads in a circumstance under which
they were a very poor choice), so Harald Hoyer and myself decided that a
rewrite was in order, so I went ahead and started working on a new app
(which I have titled 'rrn') which is now near feature parity with
prcsys.
It would be very nice if the next-gen init system didn't limit itself to
the system init part but also covered session init steps. The desktop
team is dead-set against system-wide daemons?, and as a result we've
seen a flourishing of in-session "helpers", all launched in a
more-or-less hackish, racish and unsatisfactory way:
? when the state of the art for PA is to pretend it's ESD to be
launched, there are clearly huge problems in our session init handling.
? sometimes you do a ps and see stray session daemons remaining alive
hours after session close
? it's not uncommon for GNOME to start doing things before initialising
its settings, causing users to wonder why the theme or fonts change 3s
after starting an app
? etc

Session init and system init seem very similar problems to me, and if we
ever want to go early login like windows, we'll need to coordinate
session init steps with the system init steps not finished yet. Of
course there are complications (you need to merge system presets and
user preferences, take the kind of login? and DE into account) but the
similarities should far outweight the differences.

Regards,

? with some good and a lot of bad (IMHO) reasons
? is it so different from classical system init levels?
--
Nicolas Mailhot
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Ceci est une partie de message
=?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?=
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080106/523893a4/attachment.bin
Casey Dahlin
2008-01-06 11:14:27 UTC
Permalink
Post by Nicolas Mailhot
Post by Casey Dahlin
There's a few issues with prcsys internally that make it very difficult
to add the features we want (dbus etc), as well as some not-so-trivial
implementation issues (use of pthreads in a circumstance under which
they were a very poor choice), so Harald Hoyer and myself decided that a
rewrite was in order, so I went ahead and started working on a new app
(which I have titled 'rrn') which is now near feature parity with
prcsys.
It would be very nice if the next-gen init system didn't limit itself to
the system init part but also covered session init steps. The desktop
team is dead-set against system-wide daemons?, and as a result we've
seen a flourishing of in-session "helpers", all launched in a
? when the state of the art for PA is to pretend it's ESD to be
launched, there are clearly huge problems in our session init handling.
? sometimes you do a ps and see stray session daemons remaining alive
hours after session close
? it's not uncommon for GNOME to start doing things before initialising
its settings, causing users to wonder why the theme or fonts change 3s
after starting an app
? etc
Session init and system init seem very similar problems to me, and if we
ever want to go early login like windows, we'll need to coordinate
session init steps with the system init steps not finished yet. Of
course there are complications (you need to merge system presets and
user preferences, take the kind of login? and DE into account) but the
similarities should far outweight the differences.
Regards,
? with some good and a lot of bad (IMHO) reasons
? is it so different from classical system init levels?
I'd be happy to take a look at this problem once we've gotten the core
problem out of the way.

PA in particular really could do to be a system wide service. What if we
activated GDM greeter noises?

--CJD
Linus Walleij
2008-01-06 12:48:50 UTC
Permalink
Post by Nicolas Mailhot
It would be very nice if the next-gen init system didn't limit itself to
the system init part but also covered session init steps.
AFAICS this is what Scott wants to do with "InitKit", briefly outlined
here:
http://lists.freedesktop.org/archives/initkit/2007-December/000000.html

The problem is that right now this seems to be a 1-person project so let's
see if it's catching on.

Linus
Leszek Matok
2008-01-06 12:27:14 UTC
Permalink
Post by Casey Dahlin
rewrite was in order, so I went ahead and started working on a new app
(which I have titled 'rrn') which is now near feature parity with
prcsys.
"rrn" is an old fork of "rn", a usenet reader. Besides, please, don't call
features which you will be advertising as breakthroughs in future Fedora
versions with onomatopoeias emulating my stomach :)

Lam
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080106/e2650026/attachment.bin
Casey Dahlin
2008-01-06 20:14:30 UTC
Permalink
Post by Leszek Matok
Post by Casey Dahlin
rewrite was in order, so I went ahead and started working on a new app
(which I have titled 'rrn') which is now near feature parity with
prcsys.
"rrn" is an old fork of "rn", a usenet reader. Besides, please, don't call
features which you will be advertising as breakthroughs in future Fedora
versions with onomatopoeias emulating my stomach :)
Lam
It stands for "Resolve/Run/Notify" in this context, but duly noted. I
will attempt to spare your usenet reader, and your abdomen, of any
infringement.

--CJD
Jonathan Underwood
2008-01-06 17:25:55 UTC
Permalink
Post by Casey Dahlin
Post by Jonathan Underwood
Post by Casey Dahlin
There is work being done in this area for Fedora as well. I've been
working on a parallel booting system for us that also will provide dbus
notifications for the starting of various services. (Hopefully I'll be
leading a hackfest at FUDCon in a few days to get a few more eyes and
hands on the code).
Other than compatability with sysvinit scripts, why was upstart dismissed?
Many options were looked at during the last iteration of this
discussion, and by my understanding prcsys was the one that won out. See
http://fedoraproject.org/wiki/FCNewInit/RC?highlight=%28fcnewinit%29
That's a shame, because the roadmap for upstart includes better
sysvinit compatibility, dbus support for IPC, service mangement et.
See:

https://lists.ubuntu.com/archives/upstart-devel/2007-October/000468.html

It seems to me that many of the reasons for dismissing upstart in
favour of a write-from-scratch have since disappeared.


J.
Yaakov Nemoy
2008-01-06 05:29:58 UTC
Permalink
Post by Casey Dahlin
there may be an advantage. I don't know if I like python in particular
for this roll (maybe something like Haskell that has proven to be light
and quick as well as nicer to deal with) but python is very entrenched
+1 million :D

I'm actually very interested.

-Yaakov
Casey Dahlin
2008-01-06 06:17:02 UTC
Permalink
Post by Yaakov Nemoy
Post by Casey Dahlin
there may be an advantage. I don't know if I like python in particular
for this roll (maybe something like Haskell that has proven to be light
and quick as well as nicer to deal with) but python is very entrenched
+1 million :D
I'm actually very interested.
-Yaakov
Heh. Haskell is still on my "need to learn" list. A recent glance at the
programming language shootout bumped it up a bit :)

Rewriting a few init scripts or adding LSB headers could be part of the
hackfest.

On that note, is there a packaging requirement now that any new init
scripts must have LSB headers? Should there be?

--CJD
Jason L Tibbitts III
2008-01-07 03:19:27 UTC
Permalink
CD> On that note, is there a packaging requirement now that any new
CD> init scripts must have LSB headers?

Not at this time.

CD> Should there be?

Well, a while back a pile of bugs were filed against everything with
an initscript, people set out to fix their packages and found very
poor documentation and weird corner cases but not a whole lot of
answers to their questions.

If that can be avoided, then fine. But just making up rules with
nothing to back it up doesn't really work well. Not to mention that
since initscripts are interdependent, there's actually an ordering
that must be followed when fixing them. For example, if I require an
SMTP daemon to be running before I can start, what exactly does my
initscript use for Required-Start:? I can't know that until at least
sendmail and preferably all of sendmail, exim and postfix are fixed
first.

- J<
Casey Dahlin
2008-01-07 03:33:09 UTC
Permalink
Post by Jason L Tibbitts III
CD> On that note, is there a packaging requirement now that any new
CD> init scripts must have LSB headers?
Not at this time.
CD> Should there be?
Well, a while back a pile of bugs were filed against everything with
an initscript, people set out to fix their packages and found very
poor documentation and weird corner cases but not a whole lot of
answers to their questions.
If that can be avoided, then fine. But just making up rules with
nothing to back it up doesn't really work well. Not to mention that
since initscripts are interdependent, there's actually an ordering
that must be followed when fixing them. For example, if I require an
SMTP daemon to be running before I can start, what exactly does my
initscript use for Required-Start:? I can't know that until at least
sendmail and preferably all of sendmail, exim and postfix are fixed
first.
- J<
All very true. For packages providing anonymous services we should
probably have a generic provide (such as "$syslog" for the system log,
and other such services). This could be standardized before any package
changes, allowing the packages to be updated in any order (at the moment
the validity of these dependencies doesn't matter much, as the scripts
currently shipping with Fedora attest, so if the dependencies are broken
because the target service hasn't put the correct provides in place, we
can survive). I'm sure there are more questions. There is a wiki page
covering the specifics of how Fedora intends to implement these headers:

http://fedoraproject.org/wiki/FCNewInit/Initscripts?highlight=%28fcnewinit%29

perhaps we should put effort toward maintaining it and making it
unambiguous.

I had some questions about the practicality of Required-Stop: myself.
Should we really refuse to kill a service based on a dependency? Killing
it anyway can't be worse than the angry kill -9 which inevitably follows.
Dimi Paun
2008-01-06 06:20:47 UTC
Permalink
Post by Casey Dahlin
As for rewriting some of the scripts themselves in a non-bash
language, there may be an advantage. I don't know if I like python in
particular for this roll (maybe something like Haskell that has proven
to be light and quick as well as nicer to deal with) but python is
Oh please, I hope you're not being serious! I mean Haskell is cool
and all, but it is rather obscure for the vast majority of people,
and the last thing we need is yet another strange language mixed in
such a critical part of the system.

The entire point of having a scripting language in the init scripts
is for the admins to have a chance of making small adjustments easily.
If we use Haskell, we lose 99.99% of the population right there.
We might as well just code them in C: that would provide for faster
start-up AND it will be accessible to most people.
--
Dimi Paun <dimi at lattica.com>
Lattica, Inc.
Casey Dahlin
2008-01-06 06:22:58 UTC
Permalink
Post by Dimi Paun
Post by Casey Dahlin
As for rewriting some of the scripts themselves in a non-bash
language, there may be an advantage. I don't know if I like python in
particular for this roll (maybe something like Haskell that has proven
to be light and quick as well as nicer to deal with) but python is
Oh please, I hope you're not being serious! I mean Haskell is cool
and all, but it is rather obscure for the vast majority of people,
and the last thing we need is yet another strange language mixed in
such a critical part of the system.
The entire point of having a scripting language in the init scripts
is for the admins to have a chance of making small adjustments easily.
If we use Haskell, we lose 99.99% of the population right there.
We might as well just code them in C: that would provide for faster
start-up AND it will be accessible to most people.
I don't know that I find Haskell to be that obscure, but point taken.

--CJD
Yaakov Nemoy
2008-01-06 15:52:57 UTC
Permalink
Post by Dimi Paun
Oh please, I hope you're not being serious! I mean Haskell is cool
and all, but it is rather obscure for the vast majority of people,
and the last thing we need is yet another strange language mixed in
such a critical part of the system.
The entire point of having a scripting language in the init scripts
is for the admins to have a chance of making small adjustments easily.
If we use Haskell, we lose 99.99% of the population right there.
We might as well just code them in C: that would provide for faster
start-up AND it will be accessible to most people.
It is obscure, and I agree it's very hard to justify putting more
languages in the mix.

Haskell has a few advantages over C. It's very easy to write a
wrapper in Haskell that checks if file 'foo' is newer than the binary
that it's trying to execute, and if so, to compile the new file, load
up the new binary and keep running. You can also have it redirect any
error output anywhere you like. The difference between Haskell and
Python is that instead of calling an interpreter that byte-compiles a
script and runs the byte-compiled code in a VM, it calls a compiler to
convert the Haskell into C-- code, and then calls gcc to compile that
into native code. (At least that's what GHC does.) This 'just in
time' compiling is slow the first time, but cached conveniently by the
file system for each subsequent run. XMonad takes full advantage of
this architecture so that you can literally edit your configuration,
press a key command and see your new config loaded in real time.

There are other aspects to the VM vs native machine code arguement
that are still important. For example, getting to a 'prompt' in the
vm on top of a running process is much harder and requires more boiler
plate in Haskell, whereas Python has it embedded in its standard libs.

-Yaakov
Nils Philippsen
2008-01-06 19:09:50 UTC
Permalink
Post by Yaakov Nemoy
It is obscure, and I agree it's very hard to justify putting more
languages in the mix.
Haskell has a few advantages over C. It's very easy to write a
wrapper in Haskell that checks if file 'foo' is newer than the binary
that it's trying to execute, and if so, to compile the new file, load
up the new binary and keep running.
You would have to have very good arguments to require a compiler
collection just to boot a system.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little
Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
Yaakov Nemoy
2008-01-06 19:33:39 UTC
Permalink
Post by Nils Philippsen
Post by Yaakov Nemoy
Haskell has a few advantages over C. It's very easy to write a
wrapper in Haskell that checks if file 'foo' is newer than the binary
that it's trying to execute, and if so, to compile the new file, load
up the new binary and keep running.
You would have to have very good arguments to require a compiler
collection just to boot a system.
Which are the same requirements you would have to propose to put a
just in time compiler and virtual machine. I stated before, the only
way to know if this kind of a system is ultimately faster is to
actually implement it fully, and benchmark it.

In this case, the argument would be to use Haskell to define init
scripts, and possibly other aspects of the boot process. You gain the
power Haskell provides. You lose the advantage of a well known
language.

-Yaakov
Enrico Scholz
2008-01-06 10:36:03 UTC
Permalink
Post by Casey Dahlin
There is work being done in this area for Fedora as well. I've been
working on a parallel booting system for us that also will provide
dbus notifications for the starting of various services.
mmh... does this mean, Fedora will use its own proprietary initsystem
and ignores the existing ones (initng, upstart)?
Post by Casey Dahlin
As for rewriting some of the scripts themselves in a non-bash
language,
An initsystem which requires depbloat like python or perl is completely
unacceptably.


Enrico
Casey Dahlin
2008-01-06 11:07:53 UTC
Permalink
Post by Enrico Scholz
Post by Casey Dahlin
There is work being done in this area for Fedora as well. I've been
working on a parallel booting system for us that also will provide
dbus notifications for the starting of various services.
mmh... does this mean, Fedora will use its own proprietary initsystem
and ignores the existing ones (initng, upstart)?
This has been answered here already.
Post by Enrico Scholz
Post by Casey Dahlin
As for rewriting some of the scripts themselves in a non-bash
language,
An initsystem which requires depbloat like python or perl is completely
unacceptably.
Well, I'm convinced :/
Post by Enrico Scholz
Enrico
Yaakov Nemoy
2008-01-06 15:44:58 UTC
Permalink
On Jan 6, 2008 5:36 AM, Enrico Scholz
Post by Enrico Scholz
An initsystem which requires depbloat like python or perl is completely
unacceptably.
Bash is not depbloat, but bash + awk + grep + any other userspace
tools shows alot of dep bloat. Since python is installed on most
machines, and almost always putting at least some code in shared
memory space, asking for Python in particular is not unreasonable. If
Casey had mentioned Common Lisp, I would argue the same thing.

Since depbloat should be a non-issue in this case, I think we really
need to measure how many kbytes are read on startup, and how long it
takes vs how long running python would take and how many kbytes that
brings in. Of course a hybrid system would take twice as long, so
it's a rather time consuming experiment just to shave a few seconds.

-Yaakov
Patrice Dumas
2008-01-06 15:49:35 UTC
Permalink
Post by Yaakov Nemoy
On Jan 6, 2008 5:36 AM, Enrico Scholz
Post by Enrico Scholz
An initsystem which requires depbloat like python or perl is completely
unacceptably.
Bash is not depbloat, but bash + awk + grep + any other userspace
tools shows alot of dep bloat.
bash + awk + grep is very small, not to mention that it is needed by
a lot of other softwares.
Post by Yaakov Nemoy
Since python is installed on most
machines, and almost always putting at least some code in shared
memory space, asking for Python in particular is not unreasonable. If
python is not installed in chroot rpm installs (in general) and
also may not be installed on servers.
Post by Yaakov Nemoy
Since depbloat should be a non-issue in this case, I think we really
In my opinion it is an issue in some use cases, but should not prevent
from using an init system written in python. In fact I think that the
programming language of an init system is not an issue at all.

--
Pat
Enrico Scholz
2008-01-06 16:47:57 UTC
Permalink
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 480 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080106/efbcad8f/attachment.bin
Matthew Miller
2008-01-06 19:42:38 UTC
Permalink
Since python is installed on most machines
We must have different definitions of "most". My one is ">50%" and your
Are you seriously suggesting that less than 50% of Fedora systems have yum
installed?
--
Matthew Miller mattdm at mattdm.org <http://mattdm.org/>
Boston University Linux ------> <http://linux.bu.edu/>
Enrico Scholz
2008-01-06 20:14:52 UTC
Permalink
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 480 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080106/e153bed5/attachment.bin
Daniel P. Berrange
2008-01-06 20:19:50 UTC
Permalink
Post by Matthew Miller
Since python is installed on most machines
We must have different definitions of "most". My one is ">50%" and your
Are you seriously suggesting that less than 50% of Fedora systems have yum
installed?
If you think of a traditional OS install then yum will be common. If you
think of Fedora as a base for creating black box appliances, then yum may
well not be installed at all - whole appliance upgrades are done rather
than individual packages. Likewise if you deploy Fedora in 'stateless'
mode, you're not upgrading packages on individual machines you are instead
generating a new master image. So again yum will not be there.

Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Yaakov Nemoy
2008-01-06 20:32:16 UTC
Permalink
Post by Daniel P. Berrange
If you think of a traditional OS install then yum will be common. If you
think of Fedora as a base for creating black box appliances, then yum may
well not be installed at all - whole appliance upgrades are done rather
than individual packages. Likewise if you deploy Fedora in 'stateless'
mode, you're not upgrading packages on individual machines you are instead
generating a new master image. So again yum will not be there.
Neither will Haskell, but in building one of these appliance images,
you could require Haskell as a build time requirement. Theoretically,
you could require a python compiler as well, but I don't know how far
those have come along.

-Yaakov
Toshio Kuratomi
2008-01-06 20:48:37 UTC
Permalink
Post by Yaakov Nemoy
Post by Daniel P. Berrange
If you think of a traditional OS install then yum will be common. If you
think of Fedora as a base for creating black box appliances, then yum may
well not be installed at all - whole appliance upgrades are done rather
than individual packages. Likewise if you deploy Fedora in 'stateless'
mode, you're not upgrading packages on individual machines you are instead
generating a new master image. So again yum will not be there.
Neither will Haskell, but in building one of these appliance images,
you could require Haskell as a build time requirement. Theoretically,
you could require a python compiler as well, but I don't know how far
those have come along.
pypy is doing pretty well but we're actually wandering a bit off the
point. An appliance may well want to modify the initscripts of the
services it starts. If you have to compile those with Haskell or gcc or
python then they need to be on the appliance.

A stateless box could potentially suffer from the same issue if
modifying the initscripts (temporarily) is a desired ability.

-Toshio

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: OpenPGP digital signature
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080106/9ad21730/attachment.bin
Ralf Corsepius
2008-01-06 16:48:26 UTC
Permalink
Post by Yaakov Nemoy
On Jan 6, 2008 5:36 AM, Enrico Scholz
Post by Enrico Scholz
An initsystem which requires depbloat like python or perl is completely
unacceptably.
Bash is not depbloat, but bash + awk + grep + any other userspace
tools shows alot of dep bloat.
You will want to make yourself familiar with POSIX.
Post by Yaakov Nemoy
Since python is installed on most
machines,
Only because RH/Fedora's infrastructure forces users to install it.
Post by Yaakov Nemoy
and almost always putting at least some code in shared
memory space, asking for Python in particular is not unreasonable.
I could not disagree more - To me any init-script system requiring
anything outside of what POSIX requires is a mis-conception and flawed
design.

Ralf
Nils Philippsen
2008-01-06 19:31:49 UTC
Permalink
Post by Ralf Corsepius
Post by Yaakov Nemoy
and almost always putting at least some code in shared
memory space, asking for Python in particular is not unreasonable.
I could not disagree more - To me any init-script system requiring
anything outside of what POSIX requires is a mis-conception and flawed
design.
How is that? I guess if we looked hard enough then we'd find that even
today we use things outside of POSIX to get the machine going, things
which we can't do without.

Just because something is an established standard, it's not necessarily
sufficient: One of the problems we have with SysV style init scripts are
the numerous forks and execs which are costly. I don't see how we could
solve that problem while using shell (or other tools included in POSIX).
Granted, we could source the init scripts from one master script to
avoid that, but then we'd be quicker in a cesspool of clashing global
variables or one badly written script killing the whole boot process
than we'd want to imagine. If I'm just being unimaginative, I'd be
interested in how this problem can be solved while limiting oneself to
what POSIX offers.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
Yaakov Nemoy
2008-01-06 19:34:23 UTC
Permalink
Post by Nils Philippsen
Post by Ralf Corsepius
Post by Yaakov Nemoy
and almost always putting at least some code in shared
memory space, asking for Python in particular is not unreasonable.
I could not disagree more - To me any init-script system requiring
anything outside of what POSIX requires is a mis-conception and flawed
design.
How is that? I guess if we looked hard enough then we'd find that even
today we use things outside of POSIX to get the machine going, things
which we can't do without.
Just because something is an established standard, it's not necessarily
sufficient: One of the problems we have with SysV style init scripts are
the numerous forks and execs which are costly. I don't see how we could
solve that problem while using shell (or other tools included in POSIX).
Granted, we could source the init scripts from one master script to
avoid that, but then we'd be quicker in a cesspool of clashing global
variables or one badly written script killing the whole boot process
than we'd want to imagine. If I'm just being unimaginative, I'd be
interested in how this problem can be solved while limiting oneself to
what POSIX offers.
Thanks, you said it far better than I could have :)

-Yaakov
Casey Dahlin
2008-01-06 19:34:14 UTC
Permalink
Post by Ralf Corsepius
Post by Yaakov Nemoy
On Jan 6, 2008 5:36 AM, Enrico Scholz
Post by Enrico Scholz
An initsystem which requires depbloat like python or perl is completely
unacceptably.
Bash is not depbloat, but bash + awk + grep + any other userspace
tools shows alot of dep bloat.
You will want to make yourself familiar with POSIX.
Post by Yaakov Nemoy
Since python is installed on most
machines,
Only because RH/Fedora's infrastructure forces users to install it.
Post by Yaakov Nemoy
and almost always putting at least some code in shared
memory space, asking for Python in particular is not unreasonable.
I could not disagree more - To me any init-script system requiring
anything outside of what POSIX requires is a mis-conception and flawed
design.
Is there a reason for this? It still sounds like a whole lot of "PURGE
THE IMPURE!!!" to me.

--CJD
Casey Dahlin
2008-01-07 03:45:16 UTC
Permalink
Post by Yaakov Nemoy
On Jan 6, 2008 5:36 AM, Enrico Scholz
Post by Enrico Scholz
An initsystem which requires depbloat like python or perl is completely
unacceptably.
Bash is not depbloat, but bash + awk + grep + any other userspace
tools shows alot of dep bloat. Since python is installed on most
machines, and almost always putting at least some code in shared
memory space, asking for Python in particular is not unreasonable. If
Casey had mentioned Common Lisp, I would argue the same thing.
Since depbloat should be a non-issue in this case, I think we really
need to measure how many kbytes are read on startup, and how long it
takes vs how long running python would take and how many kbytes that
brings in. Of course a hybrid system would take twice as long, so
it's a rather time consuming experiment just to shave a few seconds.
-Yaakov
What about busybox? What if we ran all the init scripts under busybox?
Its a shell-type environment, its world-famous for being incredibly
tiny, it could meet everything.
Kevin Kofler
2008-01-07 05:35:33 UTC
Permalink
Post by Casey Dahlin
What about busybox? What if we ran all the init scripts under busybox?
Its a shell-type environment, its world-famous for being incredibly
tiny, it could meet everything.
AFAIK, busybox still forks whereever a regular POSIX shell forks, so if the
amount of forks is the problem, AFAICT busybox will resolve absolutely nothing.

A shell which emulates POSIX process handling in-process and uses direct
builtin function calls for commands like sed rather than forking a new process
(even a new process of itself as busybox appears to be doing) could work, but
would that be maintainable? And what about parallelism: threads? Pipes and the
like would also have to be emulated by special-case code to become as efficient
as a real programming language, which would drive maintainability even further
down (imagine having to implement memory-to-memory, memory-to-file,
file-to-memory and file-to-file versions of all tools like sed, grep etc.).

Kevin Kofler
Casey Dahlin
2008-01-07 05:53:56 UTC
Permalink
Post by Kevin Kofler
Post by Casey Dahlin
What about busybox? What if we ran all the init scripts under busybox?
Its a shell-type environment, its world-famous for being incredibly
tiny, it could meet everything.
AFAIK, busybox still forks whereever a regular POSIX shell forks, so if the
amount of forks is the problem, AFAICT busybox will resolve absolutely nothing.
A shell which emulates POSIX process handling in-process and uses direct
builtin function calls for commands like sed rather than forking a new process
(even a new process of itself as busybox appears to be doing) could work, but
would that be maintainable? And what about parallelism: threads? Pipes and the
like would also have to be emulated by special-case code to become as efficient
as a real programming language, which would drive maintainability even further
down (imagine having to implement memory-to-memory, memory-to-file,
file-to-memory and file-to-file versions of all tools like sed, grep etc.).
Kevin Kofler
I'm not certain that the fork is the issue so much as the subsequent
execve. Its disk IO we're looking to reduce, and having all the commands
in one process image that is loaded at script start helps a lot.

--CJD
Enrico Scholz
2008-01-07 07:45:31 UTC
Permalink
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 480 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080107/4eb626d6/attachment.bin
Casey Dahlin
2008-01-07 08:14:39 UTC
Permalink
Post by Kevin Kofler
A shell which emulates POSIX process handling in-process and uses
direct builtin function calls for commands like sed [...] Pipes and
the like would also have to be emulated
For what do you need 'sed' or pipes to start/stop a daemon?
Enrico
It appears at least 13 times in our current init system.
Enrico Scholz
2008-01-07 09:44:31 UTC
Permalink
Post by Casey Dahlin
Post by Kevin Kofler
A shell which emulates POSIX process handling in-process and uses
direct builtin function calls for commands like sed [...] Pipes and
the like would also have to be emulated
For what do you need 'sed' or pipes to start/stop a daemon?
It appears at least 13 times in our current init system.
I think, nobody doubts that current initsystem is the worst one of
the major linux distributions. By changing paradigm from forking to
non-forking daemon you can avoid all the complicated 'stop' code;
e.g. 'start' will be

| pid = fork();
| if (pid==0) { /* ... */ execve(...); }

and 'stop' be

| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);


But python or other bloaty scripting languages are not a solution and
completely unacceptable at this place.



Enrico
Nils Philippsen
2008-01-07 10:42:58 UTC
Permalink
Post by Enrico Scholz
Post by Casey Dahlin
Post by Kevin Kofler
A shell which emulates POSIX process handling in-process and uses
direct builtin function calls for commands like sed [...] Pipes and
the like would also have to be emulated
For what do you need 'sed' or pipes to start/stop a daemon?
It appears at least 13 times in our current init system.
I think, nobody doubts that current initsystem is the worst one of
the major linux distributions.
Can the hyperbole please -- we know what we have otherwise we wouldn't
be having this discussion.
Post by Enrico Scholz
By changing paradigm from forking to
non-forking daemon you can avoid all the complicated 'stop' code;
e.g. 'start' will be
| pid = fork();
| if (pid==0) { /* ... */ execve(...); }
and 'stop' be
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
That'd be all fine and dandy for daemons that simply need to be started
without any variable options or preparations and are obliging enough to
write pid files. I don't think that e.g. the postgresql or xendomains
start scripts would fit well into that scheme.
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution and
completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think? If python
was split up in a base package that contained the directory structure,
the binaries, the (small) documentation and only basic modules (with no
additional requirements) and a package containing the rest of the
modules, much of the (dep-)bloat wouldn't be an issue if init scripts
(ehh modules) would be limited to that basic set. With the set of
modules I would choose, such a python-base package would have less
dependencies than bash and only account for roughly 2.6MB on the disk as
well -- bash takes up about 5.1MB.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
Enrico Scholz
2008-01-07 12:11:46 UTC
Permalink
Post by Nils Philippsen
Post by Enrico Scholz
I think, nobody doubts that current initsystem is the worst one of
the major linux distributions.
Can the hyperbole please -- we know what we have otherwise we wouldn't
be having this discussion.
I can not remember how long there are discussions about replacing
RH's initsystem (perhaps 5-6 years), and nothing happened. Other
distributions implemented more (upstart, syslog-ng) or less (suse,
mandriva) innovative methods in the meantime. And now, RH tries to
write its own implementation by adding some d-bus features and using
a bloaty scripting language around current initscripts?
Post by Nils Philippsen
Post by Enrico Scholz
By changing paradigm from forking to non-forking daemon you can avoid
all the complicated 'stop' code; e.g. 'start' will be
| pid = fork();
| if (pid==0) { /* ... */ execve(...); }
and 'stop' be
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
That'd be all fine and dandy for daemons that simply need to be started
how much resp. which daemons are not in this category?
Post by Nils Philippsen
without any variable options
options are trivially to solve; e.g. by an easy to parse one-option-per-line
configuration file or by 'argv = --foo --bar' stylish options in the service
description file.
Post by Nils Philippsen
or preparations
trivially to be solved by executing helper scripts instead of the daemon
itself

| #!/bin/sh
| ... do something ...
| exec <daemon> "$@"


Or, by using syslog-ng ideas of scripts (which terminate) and daemons
(which stay running), you can split complicated preparation code and
simple startup code in a way like

--- daemons/foo ---
require = scripts/foo
daemon = /usr/sbin/foo

--- scripts/foo ---
exec = /usr/libexec/foo-prepare
Post by Nils Philippsen
and are obliging enough to write pid files.
pid files are an anachronism required for initsystems with forking
daemons
Post by Nils Philippsen
I don't think that e.g. the postgresql
works perfectly in this way; e.g. I start it with

| /usr/bin/setuidgid postgres /usr/bin/postmaster -D /var/lib/pgsql/data
Post by Nils Philippsen
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution and
completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think?
Not with python or perl. Perhaps lua. But I really do not see why an
extra scripting language is required for the initsystem. sh/bash is
perfect for doing potential preparation tasks.
Post by Nils Philippsen
bash takes up about 5.1MB.
you mean the 750kb sized 'bash' binary and the >4 MB documentation plus
locale files?



Enrico
Nils Philippsen
2008-01-07 14:38:49 UTC
Permalink
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
I think, nobody doubts that current initsystem is the worst one of
the major linux distributions.
Can the hyperbole please -- we know what we have otherwise we wouldn't
be having this discussion.
I can not remember how long there are discussions about replacing
RH's initsystem (perhaps 5-6 years), and nothing happened. Other
distributions implemented more (upstart, syslog-ng) or less (suse,
Is syslog-ng really an init system (if yes, it would have a mighty
confusing name)?
Post by Enrico Scholz
mandriva) innovative methods in the meantime. And now, RH tries to
write its own implementation by adding some d-bus features and using
a bloaty scripting language around current initscripts?
Currently, there are no "innovative" init systems that can really be
called a standard. Therefore if the other available projects are found
lacking in some areas, trying to come up with something new that meets
the needs is an option that's perfectly fine with me. Mind that what
will get used (something existing or something new) is still in
discussion and not carved in stone yet, at least that's what I read from
FCNewInit[1].

[1]: http://fedoraproject.org/wiki/FCNewInit
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
By changing paradigm from forking to non-forking daemon you can avoid
all the complicated 'stop' code; e.g. 'start' will be
| pid = fork();
| if (pid==0) { /* ... */ execve(...); }
and 'stop' be
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
That'd be all fine and dandy for daemons that simply need to be started
how much resp. which daemons are not in this category?
If all prerequisites can be assumed as given, then none. I'm not
convinced that this assumption is a safe one to make.
Post by Enrico Scholz
Post by Nils Philippsen
without any variable options
options are trivially to solve; e.g. by an easy to parse one-option-per-line
configuration file or by 'argv = --foo --bar' stylish options in the service
description file.
Unless you specify that the command line options of the daemons are a
valid interface to have here, some logic to convert speaking
configuration options into command line options is needed. Read the
bzfs(6) manpage for an example with numerous, in part mutually exclusive
command line options where I'd rather give the user something else which
is easier to understand.
Post by Enrico Scholz
Post by Nils Philippsen
or preparations
trivially to be solved by executing helper scripts instead of the daemon
itself
| #!/bin/sh
| ... do something ...
There goes the advantage of not having to fork()/exec().
Post by Enrico Scholz
Post by Nils Philippsen
and are obliging enough to write pid files.
pid files are an anachronism required for initsystems with forking
daemons
There's no guarantee that the pid of the child process that the init
system forked will be the pid of the long running process whom you need
to send signals later on in order to end it. PID files are one way to
find this out halfway deterministically.
Post by Enrico Scholz
Post by Nils Philippsen
I don't think that e.g. the postgresql
works perfectly in this way; e.g. I start it with
| /usr/bin/setuidgid postgres /usr/bin/postmaster -D /var/lib/pgsql/data
This only works because you assume some things as a given. To make the
init process robust, services should check their prerequisites before
starting, or even ensure that they are met (e.g. /etc/init.d/sshd
generating host keys).
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution and
completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think?
Not with python or perl.
I've shown you the numbers. Why you still insist on it being bloated
beyond redemption is a mystery to me. It's more powerful than shell, but
that's easy since shell in itself can't do much without resorting to
external executables with all the fork()/exec() overhead involved. Being
powerful is not the same as being bloated in my book.
Post by Enrico Scholz
Perhaps lua. But I really do not see why an
extra scripting language is required for the initsystem. sh/bash is
perfect for doing potential preparation tasks.
Shell being perfect for anything beyond the most simple things is an
opinion that I can't share, even if you avail yourself of using
bashisms.
Post by Enrico Scholz
Post by Nils Philippsen
bash takes up about 5.1MB.
you mean the 750kb sized 'bash' binary and the >4 MB documentation plus
locale files?
Most of the modules contain their own documentation as well. Granted,
bash documentation is easier to strip away ;-). Anyway, my point was
that this would be small enough that bothering about the size of it
seems a waste of time.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
Enrico Scholz
2008-01-07 16:05:34 UTC
Permalink
Post by Nils Philippsen
Post by Enrico Scholz
I can not remember how long there are discussions about replacing
RH's initsystem (perhaps 5-6 years), and nothing happened. Other
distributions implemented more (upstart, syslog-ng) or less (suse,
Is syslog-ng really an init system
uups, I meant 'initng'.
Post by Nils Philippsen
Post by Enrico Scholz
options are trivially to solve; e.g. by an easy to parse
one-option-per-line configuration file or by 'argv = --foo --bar'
stylish options in the service description file.
Unless you specify that the command line options of the daemons
are a valid interface to have here, some logic to convert speaking
configuration options into command line options is needed. Read
the bzfs(6) manpage for an example with numerous, in part mutually
exclusive command line options where I'd rather give the user
something else which is easier to understand.
I think, when somebody wants to run a server, he has to understand how
it works and how to configure it. When admin can not figure out correct
cmdline options, how can he configure the server in a secure manner?
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
or preparations
trivially to be solved by executing helper scripts instead of the daemon
itself
| #!/bin/sh
| ... do something ...
There goes the advantage of not having to fork()/exec().
There is absolutely no problem with fork()/exec(); performance depends on
the program being executed. E.g. 'fnord' HTTP server which is started
(fork(2) + execve(2)) by tcpserver (an inetd like program) for every new
request is faster than apache httpd with no-pipelined requests.

When '... do something ...' is

| if test ! -e /etc/ssh/ssh_host_key/...'; then ... expensive tasks ...; fi

there will be "only" the overhead of /one/ execve("/bin/sh", ...)
('test' is a bash builtin) compared to the plain 'execve("<daemon>",
...)'. The number of fork()'s stays constant.


The real overhead is in loading '/bin/sh' and in interpreting the script.
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
and are obliging enough to write pid files.
pid files are an anachronism required for initsystems with forking
daemons
There's no guarantee that the pid of the child process that the init
system forked will be the pid of the long running process
then, the program is broken...
Post by Nils Philippsen
Post by Enrico Scholz
| /usr/bin/setuidgid postgres /usr/bin/postmaster -D /var/lib/pgsql/data
This only works because you assume some things as a given.
And these things *are* given. Current postgresql's initscript is
complicated because it contains code being executed exactly once.
Speaking in initng terms, you could write


--- daemons/postgresql ----
use = scripts/postgresql-prepare # depend on it only, when activated
daemon = postmaster -D ...

--- scripts/postgresql-prepare
exec = /usr/libexec/postgresql-prepare

--- /usr/lib/postgresql/postgresql-prepare ---
#! /bin/sh
...
initng-chkconfig scripts/postgresql-prepare off



This means, the expensive (has to spawn /bin/sh) -prepare script turns
off itself after running once.
Post by Nils Philippsen
To make the init process robust, services should check their prerequisites
before starting, or even ensure that they are met (e.g. /etc/init.d/sshd
generating host keys).
Question is where to draw the line. E.g. do you make 'rpm -V postgresql'
to verify that program is not corrupted?
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution
and completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think?
Not with python or perl.
I've shown you the numbers. Why you still insist on it being bloated
Resulting scripts will be much longer. E.g. how much lines of python
code are required for

| sed '/^foo/s!/bin!/opt!' file | tac

?

Most time in preparation code will be spent in "payload" (e.g. ssh-keygen).
Post by Nils Philippsen
beyond redemption is a mystery to me. It's more powerful than shell,
but that's easy since shell in itself can't do much without resorting
to external executables with all the fork()/exec() overhead involved.
Usually, preparation code will be executed only once. And I do not care
about whether first startup needs 40 or 20 seconds. But I care about,
whether regular startup needs 20 or 15 seconds.

And there *is* the bloat of initializing the python runtime environment
which can make this difference.
Post by Nils Philippsen
Post by Enrico Scholz
Perhaps lua. But I really do not see why an extra scripting language
is required for the initsystem. sh/bash is perfect for doing potential
preparation tasks.
Shell being perfect for anything beyond the most simple things is an
opinion that I can't share, even if you avail yourself of using
bashisms.
What are you missing specifically?



Enrico
Nils Philippsen
2008-01-07 17:27:35 UTC
Permalink
Post by Enrico Scholz
Post by Nils Philippsen
Unless you specify that the command line options of the daemons
are a valid interface to have here, some logic to convert speaking
configuration options into command line options is needed. Read
the bzfs(6) manpage for an example with numerous, in part mutually
exclusive command line options where I'd rather give the user
something else which is easier to understand.
I think, when somebody wants to run a server, he has to understand how
it works and how to configure it. When admin can not figure out correct
cmdline options, how can he configure the server in a secure manner?
Along that line, everybody should be able to run configure && make &&
make install and wrap their own packages, so why should we bother ;-)?
Seriously, I can cope with command line arguments and still like
sysconfig files that are more understandable than just plain
"OPTS='--foo -x=y -a'". I'm happy if I get things done without having to
read the documentation for the common case. I'm not saying admins
shouldn't be able to influence the cmd line options directly if they
wish.
Post by Enrico Scholz
The real overhead is in loading '/bin/sh' and in interpreting the script.
Agreed, I got this wrong.
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
and are obliging enough to write pid files.
pid files are an anachronism required for initsystems with forking
daemons
There's no guarantee that the pid of the child process that the init
system forked will be the pid of the long running process
then, the program is broken...
I don't think that there's a standard -- de-facto or written down -- that mandates this.
Post by Enrico Scholz
Post by Nils Philippsen
To make the init process robust, services should check their prerequisites
before starting, or even ensure that they are met (e.g. /etc/init.d/sshd
generating host keys).
Question is where to draw the line. E.g. do you make 'rpm -V postgresql'
to verify that program is not corrupted?
You know what I mean -- verifying the package integrity is just silly.
Post by Enrico Scholz
Resulting scripts will be much longer. E.g. how much lines of python
code are required for
| sed '/^foo/s!/bin!/opt!' file | tac
?
Where would you find such a line in an init script? I could construe and
equally short line in python that would take much more to be implemented
in shell and tools. That proves nothing.
Post by Enrico Scholz
Most time in preparation code will be spent in "payload" (e.g. ssh-keygen).
Post by Nils Philippsen
beyond redemption is a mystery to me. It's more powerful than shell,
but that's easy since shell in itself can't do much without resorting
to external executables with all the fork()/exec() overhead involved.
Usually, preparation code will be executed only once. And I do not care
about whether first startup needs 40 or 20 seconds. But I care about,
whether regular startup needs 20 or 15 seconds.
And there *is* the bloat of initializing the python runtime environment
which can make this difference.
The thing with the python runtime is that you'd only need to initialize
it once. Thanks to proper use of namespaces and exceptions one actually
can try to load other python modules into the same address space and not
make the whole process unstable. An empty python script takes between
10ms and 30ms start to finish on my machine here. If that initialization
time is taken only once, that's unlikely to be a bottleneck.
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
Perhaps lua. But I really do not see why an extra scripting language
is required for the initsystem. sh/bash is perfect for doing potential
preparation tasks.
Shell being perfect for anything beyond the most simple things is an
opinion that I can't share, even if you avail yourself of using
bashisms.
What are you missing specifically?
Powerful string ops come to mind, built-in regular expressions or
exceptions. I can get by without, but once you're used to it ;-).

Note that I'm not necessarily favouring a python-based init tool, I'm
just not buying your arguments against it.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
Les Mikesell
2008-01-07 17:49:56 UTC
Permalink
Post by Nils Philippsen
Post by Enrico Scholz
I think, when somebody wants to run a server, he has to understand how
it works and how to configure it. When admin can not figure out correct
cmdline options, how can he configure the server in a secure manner?
Along that line, everybody should be able to run configure && make &&
make install and wrap their own packages, so why should we bother ;-)?
Seriously, I can cope with command line arguments and still like
sysconfig files that are more understandable than just plain
"OPTS='--foo -x=y -a'". I'm happy if I get things done without having to
read the documentation for the common case. I'm not saying admins
shouldn't be able to influence the cmd line options directly if they
wish.
If you change that to some abstraction that you think is easier to
understand, how do you propose (a) that sysadmins that already knew the
real options should deal with the now confusing abstraction and (b) that
the abstraction (and its documentation) always stays in sync with
upstream changes/additions to the underlying program's options? It is
already fairly messy trying to track what options have been moved to new
locations under /etc/sysconfig in fedora/RH boxes and which are still in
their normal locations.
--
Les Mikesell
lesmikesell at gmail.com
Nils Philippsen
2008-01-08 10:26:10 UTC
Permalink
Post by Les Mikesell
Post by Nils Philippsen
Post by Enrico Scholz
I think, when somebody wants to run a server, he has to understand how
it works and how to configure it. When admin can not figure out correct
cmdline options, how can he configure the server in a secure manner?
Along that line, everybody should be able to run configure && make &&
make install and wrap their own packages, so why should we bother ;-)?
Seriously, I can cope with command line arguments and still like
sysconfig files that are more understandable than just plain
"OPTS='--foo -x=y -a'". I'm happy if I get things done without having to
read the documentation for the common case. I'm not saying admins
shouldn't be able to influence the cmd line options directly if they
wish.
If you change that to some abstraction that you think is easier to
understand, how do you propose (a) that sysadmins that already knew the
real options should deal with the now confusing abstraction
E.g. like with the old /etc/sysconfig/hdparm, you could use "speaking
options" but have an "HDPARM_OPTS" variable (or some name like that)
which would just passed on the command line.
Post by Les Mikesell
and (b) that
the abstraction (and its documentation) always stays in sync with
upstream changes/additions to the underlying program's options? It is
already fairly messy trying to track what options have been moved to new
locations under /etc/sysconfig in fedora/RH boxes and which are still in
their normal locations.
That's the job of the maintainer of the concerned package: to ensure
that sysconfig options he introduced get mapped to the correct set of
command line arguments. If those change, the mapping has to change.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
Les Mikesell
2008-01-08 13:59:52 UTC
Permalink
Post by Nils Philippsen
Post by Les Mikesell
Post by Nils Philippsen
Post by Enrico Scholz
I think, when somebody wants to run a server, he has to understand how
it works and how to configure it. When admin can not figure out correct
cmdline options, how can he configure the server in a secure manner?
Along that line, everybody should be able to run configure && make &&
make install and wrap their own packages, so why should we bother ;-)?
Seriously, I can cope with command line arguments and still like
sysconfig files that are more understandable than just plain
"OPTS='--foo -x=y -a'". I'm happy if I get things done without having to
read the documentation for the common case. I'm not saying admins
shouldn't be able to influence the cmd line options directly if they
wish.
If you change that to some abstraction that you think is easier to
understand, how do you propose (a) that sysadmins that already knew the
real options should deal with the now confusing abstraction
E.g. like with the old /etc/sysconfig/hdparm, you could use "speaking
options" but have an "HDPARM_OPTS" variable (or some name like that)
which would just passed on the command line.
And how is someone supposed to know that?
Post by Nils Philippsen
Post by Les Mikesell
and (b) that
the abstraction (and its documentation) always stays in sync with
upstream changes/additions to the underlying program's options? It is
already fairly messy trying to track what options have been moved to new
locations under /etc/sysconfig in fedora/RH boxes and which are still in
their normal locations.
That's the job of the maintainer of the concerned package: to ensure
that sysconfig options he introduced get mapped to the correct set of
command line arguments. If those change, the mapping has to change.
I think you are missing my point. I run an operating system in order to
run programs I add myself, some of which supply init scripts which I
expect to continue to run from one version to the next with no or minor
changes. Changing the way a sysv-like init service runs would be sort
of like the C language changing its keywords. Changing the
system-supplied scripts to do things more efficiently is one thing;
breaking expected system behavior is something else.
--
Les Mikesell
lesmikesell at gmail.com
Enrico Scholz
2008-01-07 18:27:53 UTC
Permalink
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
There's no guarantee that the pid of the child process that the init
system forked will be the pid of the long running process
then, the program is broken...
I don't think that there's a standard -- de-facto or written down -- that mandates this.
sorry, but a design like

parent
|-- child0
`- child1

where signals must be sent to child0 to control parent + child1 smells
somehow broken.
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
To make the init process robust, services should check their
prerequisites before starting, or even ensure that they are met
(e.g. /etc/init.d/sshd generating host keys).
Question is where to draw the line. E.g. do you make 'rpm -V postgresql'
to verify that program is not corrupted?
You know what I mean
no
Post by Nils Philippsen
Post by Enrico Scholz
Resulting scripts will be much longer. E.g. how much lines of python
code are required for
| sed '/^foo/s!/bin!/opt!' file | tac
Where would you find such a line in an init script?
Does it look so uncommon? 'sed' is used very often, pipes too. 'tac'
can be there too, e.g. with a trailing 'sed "1p;d"'
Post by Nils Philippsen
Post by Enrico Scholz
What are you missing specifically?
Powerful string ops come to mind,
which string ops other than ${..##..} + ${..%%..} do you need in
initscripts?
Post by Nils Philippsen
built-in regular expressions or
where do you need regexps in initscripts?
Post by Nils Philippsen
exceptions
set -e




Enrico
Kevin Kofler
2008-01-08 03:33:26 UTC
Permalink
Post by Enrico Scholz
Does it look so uncommon? 'sed' is used very often, pipes too. 'tac'
can be there too, e.g. with a trailing 'sed "1p;d"'
For what do you need 'sed' or pipes to start/stop a daemon?
So you're contradicting yourself.

Kevin Kofler
Enrico Scholz
2008-01-08 07:37:00 UTC
Permalink
Post by Kevin Kofler
Post by Enrico Scholz
Does it look so uncommon? 'sed' is used very often, pipes too. 'tac'
can be there too, e.g. with a trailing 'sed "1p;d"'
For what do you need 'sed' or pipes to start/stop a daemon?
So you're contradicting yourself.
no; latter was about starting/stopping a daemon. First one was done in
a discussion about preparation scripts (which are usually not needed).


Enrico
Nils Philippsen
2008-01-08 10:21:05 UTC
Permalink
[...]
Post by Enrico Scholz
sorry, but a design like
parent
|-- child0
`- child1
where signals must be sent to child0 to control parent + child1 smells
somehow broken.
I was more thinking along the lines of:

1. process 1 (p1) prepares the service
2. when finished, p1 forks p2 which will be the long-runner, controlling
any eventual children
3. p1 exits

This seems to be a standard procedure for self-detaching daemons to me.
The caller can't know or even reliably guess the pid of p2.
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
To make the init process robust, services should check their
prerequisites before starting, or even ensure that they are met
(e.g. /etc/init.d/sshd generating host keys).
Question is where to draw the line. E.g. do you make 'rpm -V postgresql'
to verify that program is not corrupted?
You know what I mean
no
You're intentionally ignoring what I mean. I could throw around terms
like "common sense", but there's a lot of dissent about what that
means ;-).
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
Resulting scripts will be much longer. E.g. how much lines of python
code are required for
| sed '/^foo/s!/bin!/opt!' file | tac
Where would you find such a line in an init script?
Does it look so uncommon? 'sed' is used very often, pipes too. 'tac'
can be there too, e.g. with a trailing 'sed "1p;d"'
I can think of easier ways to extract the last line (if I'm reading your
sed correctly ;-).
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
What are you missing specifically?
Powerful string ops come to mind,
which string ops other than ${..##..} + ${..%%..} do you need in
initscripts?
I've often had to chain multiple of these to get the desired results --
but that's more a matter of convenience (perhaps my bash is just too
rusty and it can be done more elegantly). What matters more to me (this
came to mind only yesterday while driving home) is that if you do:

cat $somefile | while read line; do ... done

variables set in the while loop have no effect outside of it, or if you
want them to have effect, you have to do nasty things (or my bash is
rusty again ;-).
Post by Enrico Scholz
Post by Nils Philippsen
built-in regular expressions or
where do you need regexps in initscripts?
To (more) easily extract certain part of strings for example, instead of
using multiple removal of heading and trailing characters. To find out
whether a line matches a certain pattern, think of your "sed
'/^foo/...'" construct above.
Post by Enrico Scholz
Post by Nils Philippsen
exceptions
set -e
Throwing exceptions is one thing, but you should be able to catch them,
too. E.g.:

try:
# some_complicated_task
...
# problem
raise SomeException ("Something's fishy, Dave")
...
# more serious problem
raise SomeSeriousError ("Something's screwed, Dave")
...
except OSError, e:
# clean up OS error
if e.errno == ...:
...
...
except SomeException, e:
# do something else
sys.stderr.print "Warning: doing foo failed, continuing (%s)" % str (e)
except:
# panic!
sys.stderr.print "Error: %s" % str(e)
sys.exit (42)

I don't know if any of this is necessary for init code, but these are
some things that I miss or find cumbersome in shell.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
Enrico Scholz
2008-01-08 12:10:27 UTC
Permalink
Post by Nils Philippsen
1. process 1 (p1) prepares the service
2. when finished, p1 forks p2 which will be the long-runner, controlling
any eventual children
3. p1 exits
This seems to be a standard procedure for self-detaching daemons to me.
That's the old forking-daemon way. Most modern initsystems work best
with non-forking daemons.
Post by Nils Philippsen
I've often had to chain multiple of these to get the desired results --
but that's more a matter of convenience (perhaps my bash is just too
rusty and it can be done more elegantly). What matters more to me (this
cat $somefile | while read line; do ... done
variables set in the while loop have no effect outside of it, or if you
this effect is known and this example can be written as

| while read line ... done < $somefile


But we are speaking about preparation scripts of daemons. Do you really
need such tasks there?
Post by Nils Philippsen
# clean up OS error
...
# do something else
sys.stderr.print "Warning: doing foo failed, continuing (%s)" % str (e)
# panic!
sys.stderr.print "Error: %s" % str(e)
sys.exit (42)
I don't know if any of this is necessary for init code, but these are
some things that I miss or find cumbersome in shell.
I do not say that shell shall be used for everything. I just say that
python is not an option for any part of sysinit which should be written
completely in C. (ba)sh plus usual POSIX/GNU tools are the optimal
choice for preparation tasks, but these are no deps of the sysinit
system itself but of the specific daemon.



Enrico
Lennart Poettering
2008-01-07 17:35:08 UTC
Permalink
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
I can not remember how long there are discussions about replacing
RH's initsystem (perhaps 5-6 years), and nothing happened. Other
distributions implemented more (upstart, syslog-ng) or less (suse,
Is syslog-ng really an init system
uups, I meant 'initng'.
Everytime I hear someone mentioning initng I get a headache.

They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"), and added the worst things they could find in other
people's software. Like the braindeadness to make everything a shared
object, including stuff like executing chdir(). Can you believe that?
They have a "plugin" to change a directory which consists of 100 lines
or code or so. Unbelievable...

Lennart
--
Lennart Poettering Red Hat, Inc.
lennart [at] poettering [dot] net ICQ# 11060553
http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Casey Dahlin
2008-01-07 17:35:39 UTC
Permalink
Post by Lennart Poettering
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
I can not remember how long there are discussions about replacing
RH's initsystem (perhaps 5-6 years), and nothing happened. Other
distributions implemented more (upstart, syslog-ng) or less (suse,
Is syslog-ng really an init system
uups, I meant 'initng'.
Everytime I hear someone mentioning initng I get a headache.
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"), and added the worst things they could find in other
people's software. Like the braindeadness to make everything a shared
object, including stuff like executing chdir(). Can you believe that?
They have a "plugin" to change a directory which consists of 100 lines
or code or so. Unbelievable...
Lennart
This is all very useful.

I am proposing a session at FUDcon to explore the options, and to
definitively pick a solution. Hopefully the following day's hackfest
will see direct effort toward implementing a solution.
Jonathan Underwood
2008-01-07 18:10:13 UTC
Permalink
Post by Casey Dahlin
I am proposing a session at FUDcon to explore the options, and to
definitively pick a solution. Hopefully the following day's hackfest
will see direct effort toward implementing a solution.
Having a read of this document proves very enlightening about the
issues that need to be addressed in designing a replacement init:


https://lists.ubuntu.com/archives/upstart-devel/2007-September/000463.html
Lennart Poettering
2008-01-07 18:12:20 UTC
Permalink
Post by Casey Dahlin
Post by Lennart Poettering
Everytime I hear someone mentioning initng I get a headache.
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"), and added the worst things they could find in other
people's software. Like the braindeadness to make everything a shared
object, including stuff like executing chdir(). Can you believe that?
They have a "plugin" to change a directory which consists of 100 lines
or code or so. Unbelievable...
Lennart
This is all very useful.
I am proposing a session at FUDcon to explore the options, and to
definitively pick a solution. Hopefully the following day's hackfest will
see direct effort toward implementing a solution.
"Definitively" picking a solution? Unfortunately I don't see that any
of the currently available init implementations get things
right in a way that we could "definitively" pick it.

The only one of the new systems that has good code is
Upstart. However, in my understanding it got everything hooked up the
wrong way round. I.e. instead of having other daemons contact Upstart
to start and stop services it itself hooks into all kind of
"events". A couple of RH and Novell people discussed that with Scott
at this years GUADEC conference a while back. I think we managed to
convince him that this should be changed. The result is his new
Initkit project. However, that's still in its infancy and will take
some time to be useful. This however means that now adopting Upstart
would be investing in a project that's going to be replaced soonishly
anyway. Also, Ubuntu uses Upstart mostly in SysV compatibility mode
right now.

So, in short: initng is a joke, initkit not ready yet, upstart a bit of
a moving target that's going to be replaced soon anyway. The other
systems seem to be too simple (minit, runit) or totally un-Linuxish
(SMF, launchd).

I think our safest bet for now is to stay with SysV but spice it up a
little bit with LSB headers to allow parallel startup, like Debian is
doing it now.

And then, let's wait what Scott comes up with in InitKit. Given that
he's a Canonical guy, and both RH and Novell engineers discussed
Upstart in lengths with him I hope that this is also the best bet to
get something done that is adopted by all "big three" distributions,
working a bit against the balkanization of Linux userspace.

Lennart
--
Lennart Poettering Red Hat, Inc.
lennart [at] poettering [dot] net ICQ# 11060553
http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Casey Dahlin
2008-01-07 19:21:28 UTC
Permalink
Post by Lennart Poettering
Post by Casey Dahlin
Post by Lennart Poettering
Everytime I hear someone mentioning initng I get a headache.
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"), and added the worst things they could find in other
people's software. Like the braindeadness to make everything a shared
object, including stuff like executing chdir(). Can you believe that?
They have a "plugin" to change a directory which consists of 100 lines
or code or so. Unbelievable...
Lennart
This is all very useful.
I am proposing a session at FUDcon to explore the options, and to
definitively pick a solution. Hopefully the following day's hackfest will
see direct effort toward implementing a solution.
"Definitively" picking a solution? Unfortunately I don't see that any
of the currently available init implementations get things
right in a way that we could "definitively" pick it.
The only one of the new systems that has good code is
Upstart. However, in my understanding it got everything hooked up the
wrong way round. I.e. instead of having other daemons contact Upstart
to start and stop services it itself hooks into all kind of
"events". A couple of RH and Novell people discussed that with Scott
at this years GUADEC conference a while back. I think we managed to
convince him that this should be changed. The result is his new
Initkit project. However, that's still in its infancy and will take
some time to be useful. This however means that now adopting Upstart
would be investing in a project that's going to be replaced soonishly
anyway. Also, Ubuntu uses Upstart mostly in SysV compatibility mode
right now.
So, in short: initng is a joke, initkit not ready yet, upstart a bit of
a moving target that's going to be replaced soon anyway. The other
systems seem to be too simple (minit, runit) or totally un-Linuxish
(SMF, launchd).
I think our safest bet for now is to stay with SysV but spice it up a
little bit with LSB headers to allow parallel startup, like Debian is
doing it now.
And then, let's wait what Scott comes up with in InitKit. Given that
he's a Canonical guy, and both RH and Novell engineers discussed
Upstart in lengths with him I hope that this is also the best bet to
get something done that is adopted by all "big three" distributions,
working a bit against the balkanization of Linux userspace.
Lennart
Fine. Let's commit to that outright, and not let ourselves sit on our
asses for another 5 years while everything blows by. I'm pushing this
forward because I'm sick of seeing the action being taken on the weakest
point of our distro consisting of a bunch of stale wiki pages.

My point in saying "definitively" is that this will not be another
mailing list thread where everyone shouts back and forth until they get
bored and then everyone forgets about the issue. I mean to ensure we
pick a goal, agree to stop bitching about it, and actually execute on it.

rrn or whatever it ends up being called exists because I asked Harald
Hoyer what needed to exist for Fedora's init system to be healed, and
then I went and made it exist. Now we don't like that solution anymore.
Great. Pick another one, and ensure it begins happening as fast as
possible. Or don't. I'll do all the work. Just don't complain when its
finished. Commit, and be done with it. It is time to set a terminal date
beyond which discussion of other solutions is no longer welcome. When we
are interested only in the implementation of that which has already been
decided. Source code or GTFO.

Fedora 9 is probably going to slip without a feature that should have
been in Fedora 8. We're supposed to be innovators, and look at how much
nothing we've done on this issue. This is absurd. I'm saying we end it now.

--CJD
Bill Nottingham
2008-01-07 19:24:32 UTC
Permalink
Post by Lennart Poettering
I think our safest bet for now is to stay with SysV but spice it up a
little bit with LSB headers to allow parallel startup, like Debian is
doing it now.
The problem with how the LSB headers is that as it currently stands
(i.e., not counting prcsys or whatever), they're evaluated at script
install time, and to *correctly* handle them, you need to do it at
runtime, with information that's not in the scripts themselves. So,
to do this right, you need something new to at least replace /etc/rc.

Whether that's worth the effort is another issue.

Bill
Casey Dahlin
2008-01-07 19:30:41 UTC
Permalink
Post by Bill Nottingham
Post by Lennart Poettering
I think our safest bet for now is to stay with SysV but spice it up a
little bit with LSB headers to allow parallel startup, like Debian is
doing it now.
The problem with how the LSB headers is that as it currently stands
(i.e., not counting prcsys or whatever), they're evaluated at script
install time, and to *correctly* handle them, you need to do it at
runtime, with information that's not in the scripts themselves. So,
to do this right, you need something new to at least replace /etc/rc.
Whether that's worth the effort is another issue.
Bill
I have one. Its 60% done, will be ready for rawhide after Fudcon if we
decide to pursue it. See earlier in the thread.

--CJD
Bill Nottingham
2008-01-07 19:57:42 UTC
Permalink
Post by Casey Dahlin
I have one. Its 60% done, will be ready for rawhide after Fudcon if we
decide to pursue it. See earlier in the thread.
How are you handling $time, $localfs, $named, etc?

Bill
Casey Dahlin
2008-01-07 20:08:41 UTC
Permalink
Post by Bill Nottingham
Post by Casey Dahlin
I have one. Its 60% done, will be ready for rawhide after Fudcon if we
decide to pursue it. See earlier in the thread.
How are you handling $time, $localfs, $named, etc?
Bill
At the moment they are just like any other provides, so they would be
handled by a script (which will presumably just report on the status of
the given item). This will likely change, however.

--CJD
Bill Nottingham
2008-01-07 20:12:03 UTC
Permalink
Post by Casey Dahlin
Post by Bill Nottingham
How are you handling $time, $localfs, $named, etc?
At the moment they are just like any other provides, so they would be
handled by a script (which will presumably just report on the status of the
given item). This will likely change, however.
Right, that's the tricky part. $localfs leads to 'fun' loops when it's
handled as part of the script, since it needs to be added at the latest
possible point (after $network_fs, therefore in netfs). This then leads
to loops. Similarly, $named needs to be provided by named, if it's started.
Otherwise it needs to be provided by $network. Or possibly openldap, if
you're using nss_ldap in your nsswitch.conf.

Bill
Les Mikesell
2008-01-07 17:59:41 UTC
Permalink
Post by Lennart Poettering
Everytime I hear someone mentioning initng I get a headache.
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What's wrong with numerical runlevels other than Linux flavors starting
the network in level 2 instead of 3?
--
Les Mikesell
lesmikesell at gmail.com
Lennart Poettering
2008-01-07 18:26:48 UTC
Permalink
Post by Lennart Poettering
Everytime I hear someone mentioning initng I get a headache.
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What's wrong with numerical runlevels other than Linux flavors starting the
network in level 2 instead of 3?
They don't make any sense, that's wrong with them.

Why only 6 of them?

I never understood why a process like "shutdown" or "reboot" should be
considered a "level". Do you?

Nobody knows what the actually mean. They are using numeric ids for no
reason. I mean, people invented stuff like DNS for not having to
deal with random numbers that often. Why should they deal with random
numbers when dealing with init systems, then?

Almost all distributions use only 2 or 3 of them.

Their configuration is plain awful.

They're totally awkward, because there are numeric levels and the
magic runlevel S.

Also, init doesn't really have any information what service is running
in which one is not for the current runlevel. To work around that the
configuration is highly redundant with lots of symlinks.

Then, on the philosophical level, having a single set of "runlevels"
just doesn't cut it, because services these days should only be
started maybe when a certain HW is around, or when some specific user
software runs that wants to make use of it. How do runlevels fit into
that? They are too static to reflect this dynamic way of starting and
stopping services properly.

In short, they don't make any sense. They should die.

Lennart
--
Lennart Poettering Red Hat, Inc.
lennart [at] poettering [dot] net ICQ# 11060553
http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Les Mikesell
2008-01-07 19:06:59 UTC
Permalink
Post by Lennart Poettering
What's wrong with numerical runlevels other than Linux flavors starting the
network in level 2 instead of 3?
They don't make any sense, that's wrong with them.
1 as single user, 2 as multiuser makes reasonable sense, with networking
as the logical next step.
Post by Lennart Poettering
Why only 6 of them?
Until someone decided to make X something special you only needed 3
steps that you could jump among: single user, multiuser,
multiuser+network. But with a scheme based on directory names and the
inherent sort order of a wildcard expansion, there initially wasn't a
real limit.
Post by Lennart Poettering
I never understood why a process like "shutdown" or "reboot" should be
considered a "level". Do you?
runlevel 0 = quit sounds extremely logical. 6=reboot was someone just
running out of ideas.
Post by Lennart Poettering
Nobody knows what the actually mean.
Unless they read a SysV manual back when they fit in your pocket?
Post by Lennart Poettering
They are using numeric ids for no
reason.
Their sort order is well known.
Post by Lennart Poettering
I mean, people invented stuff like DNS for not having to
deal with random numbers that often. Why should they deal with random
numbers when dealing with init systems, then?
Almost all distributions use only 2 or 3 of them.
Their configuration is plain awful.
They're totally awkward, because there are numeric levels and the
magic runlevel S.
Well, yeah - but then there is q also - and necessarily so...
Post by Lennart Poettering
Also, init doesn't really have any information what service is running
in which one is not for the current runlevel.
That's the point. Init should not know anything specifically except for
the stuff it manages directly per inittab.
Post by Lennart Poettering
To work around that the
configuration is highly redundant with lots of symlinks.
Which lets it do anything it needs to do, finding out only when it needs
to know. All very good things.
Post by Lennart Poettering
Then, on the philosophical level, having a single set of "runlevels"
just doesn't cut it, because services these days should only be
started maybe when a certain HW is around, or when some specific user
software runs that wants to make use of it. How do runlevels fit into
that? They are too static to reflect this dynamic way of starting and
stopping services properly.
In short, they don't make any sense. They should die.
They make sense for what they do - otherwise how can I tell the system I
want to go down to single user mode with a clear definition of what will
still be running? And I should be able to tell it I want multiuser mode
but with no networking, but Linux distros screw that up. But init
isn't limited to those, it also does things specified in inittab. If
you want magic stuff to happen on certain events, respawn the thing that
detects the event starting at the desired runlevel and let that
program do whatever is supposed to happen next. Getty watching for a
carrier detect hardware event is the original example - and it doesn't
need to know anything except what to exec next and doesn't need to wait
around for results. Processes are supposed to be cheap enough that if
you need one you can use it - don't make init need to know anything else.
--
Les Mikesell
lesmikesell at gmail.com
Enrico Scholz
2008-01-07 18:05:12 UTC
Permalink
Post by Lennart Poettering
Everytime I hear someone mentioning initng I get a headache.
They got almost everything wrong you can get wrong in an init
system.
ack; they had bad code quality which was tried to be solved with stupid
ideas like a garbage collector, or now they are back at (ba)sh based
initscripts.

But ideas like declarative init"scripts", the use/require concept are a
real improvement compared with SysV init.
Post by Lennart Poettering
The kept the worst things from SysV (such as numerical "runlevels"), and
added the worst things they could find in other people's software. Like
the braindeadness to make everything a shared object, including stuff like
executing chdir(). Can you believe that?
Yes; its like the basearchonly plugin of yum or the tab-mix-plus plugin
of firefox. More or less trivial stuff and used by most people, but
author does not want it in code base.



Enrico
Rudolf Kastl
2008-01-08 16:18:16 UTC
Permalink
Post by Lennart Poettering
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
I can not remember how long there are discussions about replacing
RH's initsystem (perhaps 5-6 years), and nothing happened. Other
distributions implemented more (upstart, syslog-ng) or less (suse,
Is syslog-ng really an init system
uups, I meant 'initng'.
Everytime I hear someone mentioning initng I get a headache.
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What do you mean with numerical runlevels? does "default" for the
standard runlevel look "numerical" to you?

and added the worst things they could find in other
Post by Lennart Poettering
people's software. Like the braindeadness to make everything a shared
object, including stuff like executing chdir(). Can you believe that?
They have a "plugin" to change a directory which consists of 100 lines
or code or so. Unbelievable...
Lennart
--
Lennart Poettering Red Hat, Inc.
lennart [at] poettering [dot] net ICQ# 11060553
http://0pointer.net/lennart/ GnuPG 0x1A015CC4
--
fedora-devel-list mailing list
fedora-devel-list at redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Les Mikesell
2008-01-08 16:31:55 UTC
Permalink
Post by Rudolf Kastl
Post by Lennart Poettering
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What do you mean with numerical runlevels? does "default" for the
standard runlevel look "numerical" to you?
Yes, where it is defined in the 'id' line in inittab. And if it doesn't
stay that way, all of the machines I manage will crash and burn.
--
Les Mikesell
lesmikesell at gmail.com
Rudolf Kastl
2008-01-08 16:33:40 UTC
Permalink
Post by Les Mikesell
Post by Rudolf Kastl
Post by Lennart Poettering
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What do you mean with numerical runlevels? does "default" for the
standard runlevel look "numerical" to you?
Yes, where it is defined in the 'id' line in inittab. And if it doesn't
stay that way, all of the machines I manage will crash and burn.
inittab is sysV init config file... it has nothing to do with initng ;)

regards,
Rudolf Kastl
Post by Les Mikesell
--
Les Mikesell
lesmikesell at gmail.com
--
fedora-devel-list mailing list
fedora-devel-list at redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Les Mikesell
2008-01-08 16:53:43 UTC
Permalink
Post by Rudolf Kastl
Post by Les Mikesell
Post by Rudolf Kastl
Post by Lennart Poettering
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What do you mean with numerical runlevels? does "default" for the
standard runlevel look "numerical" to you?
Yes, where it is defined in the 'id' line in inittab. And if it doesn't
stay that way, all of the machines I manage will crash and burn.
inittab is sysV init config file... it has nothing to do with initng ;)
OK, that's a good reason not to use it, then.
--
Les Mikesell
lesmikesell at gmail.com
Rudolf Kastl
2008-01-08 17:37:00 UTC
Permalink
Post by Les Mikesell
Post by Rudolf Kastl
Post by Les Mikesell
Post by Rudolf Kastl
Post by Lennart Poettering
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What do you mean with numerical runlevels? does "default" for the
standard runlevel look "numerical" to you?
Yes, where it is defined in the 'id' line in inittab. And if it doesn't
stay that way, all of the machines I manage will crash and burn.
inittab is sysV init config file... it has nothing to do with initng ;)
OK, that's a good reason not to use it, then.
i dont get your logic...

kind regards,
Rudolf Kastl
Post by Les Mikesell
--
Les Mikesell
lesmikesell at gmail.com
--
fedora-devel-list mailing list
fedora-devel-list at redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Les Mikesell
2008-01-08 19:37:53 UTC
Permalink
Post by Rudolf Kastl
Post by Les Mikesell
Post by Rudolf Kastl
Post by Les Mikesell
Post by Rudolf Kastl
Post by Lennart Poettering
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What do you mean with numerical runlevels? does "default" for the
standard runlevel look "numerical" to you?
Yes, where it is defined in the 'id' line in inittab. And if it doesn't
stay that way, all of the machines I manage will crash and burn.
inittab is sysV init config file... it has nothing to do with initng ;)
OK, that's a good reason not to use it, then.
i dont get your logic...
People running fedora will expect to use sysV style init configuration
to control it.
--
Les Mikesell
lesmikesell at gmail.com
Casey Dahlin
2008-01-08 20:38:38 UTC
Permalink
Post by Les Mikesell
Post by Rudolf Kastl
Post by Les Mikesell
Post by Rudolf Kastl
Post by Les Mikesell
Post by Rudolf Kastl
Post by Lennart Poettering
They got almost everything wrong you can get wrong in an init
system. The kept the worst things from SysV (such as numerical
"runlevels"),
What do you mean with numerical runlevels? does "default" for the
standard runlevel look "numerical" to you?
Yes, where it is defined in the 'id' line in inittab. And if it doesn't
stay that way, all of the machines I manage will crash and burn.
inittab is sysV init config file... it has nothing to do with initng ;)
OK, that's a good reason not to use it, then.
i dont get your logic...
People running fedora will expect to use sysV style init configuration
to control it.
great. Maybe we should move back to KDE 1, since a lot of people got
used to that at the beginning. :( In fact we don't offer compatibility
with the abacus. I'm sure computers alienated a lot of abacus users way
back when, maybe we could do something about that.

Interfaces change. We like to keep them the same but snapping the reigns
and yelling "giddyup" just isn't a suitable control mechanism for a car.
A new init system is likely going to have properties that inittab isn't
capable of expressing. Users will have to learn. Anyone manually editing
text config files is comfortable enough to handle the changes.

--CJD
Les Mikesell
2008-01-08 20:55:34 UTC
Permalink
Post by Casey Dahlin
Interfaces change. We like to keep them the same but snapping the reigns
and yelling "giddyup" just isn't a suitable control mechanism for a car.
Great - how will you like it when the gas/brake pedals are reversed on
your next model?
Post by Casey Dahlin
A new init system is likely going to have properties that inittab isn't
capable of expressing.
What does that have to do with maintaining interfaces? If someone can't
write a backwards-compatible interface why should you trust them to be
able to improve something?
Post by Casey Dahlin
Users will have to learn.
Or not.
--
Les Mikesell
lesmikesell at gmail.com
Linus Walleij
2008-01-08 22:14:34 UTC
Permalink
People running fedora will expect to use sysV style init configuration to
control it.
Now, I think Lennart is right in pushing the concept behind Upstart and
the new InitKit, both of which break the init config paradigm and its
runlevels.

The reason was actually outlined in Miguel de Icaza's "Let's Make Unix Not
Suck" a few years back. It outlined some weaknesses of the Unix pipe and
filter and signalling system: pipes are unidirectional, data is not typed,
signals are crude in essence. Component-based thinking through CORBA led
to the invention of Bonobo, then the condensed DCOP and eventually D-Bus
which actually does the tricks most sought after: bidirectional messages
between processes, typed messages, a strict namespace, broadcast messages.

The SysVInit system currently suffers from not being able to use such a
mechanism.

Upstart solved it, basically, but has some design flaws and is used in
init-compatibility mode in Ubuntu. So now InitKit is coming along.

It's worth sacrificing runlevels to reach the next step of unsucky Unix.

POSIX does not mandate init and its runlevels, nor does the Single Unix
spec. I think there is a good reason for: it was awkward, so it wasn't
standardized. If everyone though it was a good idea they would have
standardized it back when POSIX was written. (I wasn't a member of the
committes tho, so who knows.)

Linus
Casey Dahlin
2008-01-08 22:29:42 UTC
Permalink
Post by Linus Walleij
Post by Les Mikesell
People running fedora will expect to use sysV style init
configuration to control it.
Now, I think Lennart is right in pushing the concept behind Upstart
and the new InitKit, both of which break the init config paradigm and
its runlevels.
The reason was actually outlined in Miguel de Icaza's "Let's Make Unix
Not Suck" a few years back. It outlined some weaknesses of the Unix
pipe and filter and signalling system: pipes are unidirectional, data
is not typed, signals are crude in essence. Component-based thinking
through CORBA led to the invention of Bonobo, then the condensed DCOP
bidirectional messages between processes, typed messages, a strict
namespace, broadcast messages.
The SysVInit system currently suffers from not being able to use such
a mechanism.
Upstart solved it, basically, but has some design flaws and is used in
init-compatibility mode in Ubuntu. So now InitKit is coming along.
It's worth sacrificing runlevels to reach the next step of unsucky Unix.
POSIX does not mandate init and its runlevels, nor does the Single
Unix spec. I think there is a good reason for: it was awkward, so it
wasn't standardized. If everyone though it was a good idea they would
have standardized it back when POSIX was written. (I wasn't a member
of the committes tho, so who knows.)
Linus
I agree. Also, I don't necessarily think we should wait for InitKit.
InitKit is more of a fork-and-abandon of upstart (though major changes
are in order) and I don't thing Upstart would be a bad stepping stone in
the meantime. I will talk to the developer about this. Getting features
to users sooner is a good thing :)

--CJD
Ian Burrell
2008-01-08 22:45:14 UTC
Permalink
Post by Linus Walleij
People running fedora will expect to use sysV style init configuration to
control it.
Now, I think Lennart is right in pushing the concept behind Upstart and
the new InitKit, both of which break the init config paradigm and its
runlevels.
The reason was actually outlined in Miguel de Icaza's "Let's Make Unix Not
Suck" a few years back. It outlined some weaknesses of the Unix pipe and
filter and signalling system: pipes are unidirectional, data is not typed,
signals are crude in essence. Component-based thinking through CORBA led
to the invention of Bonobo, then the condensed DCOP and eventually D-Bus
which actually does the tricks most sought after: bidirectional messages
between processes, typed messages, a strict namespace, broadcast messages.
The SysVInit system currently suffers from not being able to use such a
mechanism.
Upstart solved it, basically, but has some design flaws and is used in
init-compatibility mode in Ubuntu. So now InitKit is coming along.
It's worth sacrificing runlevels to reach the next step of unsucky Unix.
POSIX does not mandate init and its runlevels, nor does the Single Unix
spec. I think there is a good reason for: it was awkward, so it wasn't
standardized. If everyone though it was a good idea they would have
standardized it back when POSIX was written. (I wasn't a member of the
committes tho, so who knows.)
I think it is important to look what interfaces the init system
provides and which ones should be preserved. The current init has
three, /etc/inittab to determine what is done for runlevels, the
/etc/rc.d/rc symlinks to determine what services are run by
/etc/rc.d/rc, and the /etc/init.d scripts that are executed.

I think it is important that we preserve the ability to use sysv-style
init scripts. They are part of LSB. It would be lots of work to
convert everything to any new way. They are installed by third-party
packages. It might be possible to change the functions to work better
with a new system (for example by not forking when running new
programs). I think it would be acceptable if the new features
required a new file format as long as the old one is supported. The
/sbin/service program should work with both. It would be nice if
there was a program that could use the LSB metadata to configure
dependencies and other startup.

The /etc/inittab config file is specific to SysV init. Other systems
don't even have the concept of runlevels. It would be nice to
preserve the /etc/rc.d symlinks for determining which service to
start, but the standard interface is the chkconfig program. Other
Linux distributions with SysV init have the symlinks in different
places. It would be nice if there was a program that could produce
the new configuration from the old and even synchronize them after
changes to the old.

- Ian
Nicolas Mailhot
2008-01-08 22:22:28 UTC
Permalink
Post by Les Mikesell
People running fedora will expect to use sysV style init configuration
to control it.
IMHO people running Fedora (and RHEL) will gladly take any well-designed
system over what exists now, and to hell with sysV style init if that
helps making a good replacement. It's all very well to perpetuate
habits, but don't overestimate the attachment people have to the current
system.
--
Nicolas Mailhot
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Ceci est une partie de message
=?ISO-8859-1?Q?num=E9riquement?= =?ISO-8859-1?Q?_sign=E9e?=
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080108/e1ca2de6/attachment.bin
Les Mikesell
2008-01-08 23:45:27 UTC
Permalink
Post by Nicolas Mailhot
Post by Les Mikesell
People running fedora will expect to use sysV style init configuration
to control it.
IMHO people running Fedora (and RHEL) will gladly take any well-designed
system over what exists now, and to hell with sysV style init if that
helps making a good replacement. It's all very well to perpetuate
habits, but don't overestimate the attachment people have to the current
system.
But to be better, it has to do all that the old system did and more.
Would it be able to process commands to suspend-to-ram/suspend-to-disk
with sensibly sequenced operations as well as intercept the hardware
hints that these things should be done - and likewise order the
operations as it wakes up from these states?
--
Les Mikesell
lesmikesell at gmail.com
nodata
2008-01-08 18:59:56 UTC
Permalink
Post by Lennart Poettering
such as numerical "runlevels")
I like Gentoo's init scripts. Very clean, with dependencies.
Andrew Farris
2008-01-07 23:27:29 UTC
Permalink
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution
and completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think?
Not with python or perl.
I've shown you the numbers. Why you still insist on it being bloated
Resulting scripts will be much longer. E.g. how much lines of python
code are required for
| sed '/^foo/s!/bin!/opt!' file | tac
Looks like one line in python to me if you write a sed and tac replacement into
your python library. (e.g. thats not about bash vs python at all...)
--
Andrew Farris <lordmorgul at gmail.com> <ajfarris at gmail.com>
gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
---- ----
Nils Philippsen
2008-01-08 15:23:42 UTC
Permalink
Post by Andrew Farris
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution
and completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think?
Not with python or perl.
I've shown you the numbers. Why you still insist on it being bloated
Resulting scripts will be much longer. E.g. how much lines of python
code are required for
| sed '/^foo/s!/bin!/opt!' file | tac
Looks like one line in python to me if you write a sed and tac replacement into
your python library. (e.g. thats not about bash vs python at all...)
If I'm not to try to emulate the single commands, but look at the whole
job which I understand as "read file, replace first occurrence of '/bin'
with '/opt' in lines beginning with 'foo', then put out the lines in
reversed order" it could be along this (and wouldn't use any external
module or program):

print '\n'.join (reversed (map (lambda line: line[:3] == "foo" and line.replace ('/bin', '/opt', 1) or line, ''.join (open ('file').readlines ()).split ('\n'))))

To make this more readable, I'd split this up and comment it:

--- 8< ---
# read the file and split it at '\n'
lines = ''.join (open ('file').readlines ()).split ('\n')

for i in range (len (lines)):
if line[:3] == 'foo':
# replace first occurrence of /bin with /opt
lines[i] = lines[i].replace ('/bin', '/opt', 1)

print '\n'.join (reversed (lines))
--- >8 ---

Now the original sed can surely be changed that it would take more than
one line to emulate the line's function, but then I would feel
comfortable using (shipped) modules like "re" to achieve the same effect
as the shell code uses sed and tac.

Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
nodata
2008-01-08 22:21:30 UTC
Permalink
Post by Nils Philippsen
Post by Andrew Farris
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
Post by Nils Philippsen
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution
and completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think?
Not with python or perl.
I've shown you the numbers. Why you still insist on it being bloated
Resulting scripts will be much longer. E.g. how much lines of python
code are required for
| sed '/^foo/s!/bin!/opt!' file | tac
Looks like one line in python to me if you write a sed and tac replacement into
your python library. (e.g. thats not about bash vs python at all...)
If I'm not to try to emulate the single commands, but look at the whole
job which I understand as "read file, replace first occurrence of '/bin'
with '/opt' in lines beginning with 'foo', then put out the lines in
reversed order" it could be along this (and wouldn't use any external
print '\n'.join (reversed (map (lambda line: line[:3] == "foo" and line.replace ('/bin', '/opt', 1) or line, ''.join (open ('file').readlines ()).split ('\n'))))
--- 8< ---
# read the file and split it at '\n'
lines = ''.join (open ('file').readlines ()).split ('\n')
Yikes! Python just became Perl :)
Post by Nils Philippsen
# replace first occurrence of /bin with /opt
lines[i] = lines[i].replace ('/bin', '/opt', 1)
print '\n'.join (reversed (lines))
--- >8 ---
Now the original sed can surely be changed that it would take more than
one line to emulate the line's function, but then I would feel
comfortable using (shipped) modules like "re" to achieve the same effect
as the shell code uses sed and tac.
Nils
--
Nils Philippsen / Red Hat / nphilipp at redhat.com
"Those who would give up Essential Liberty to purchase a little Temporary
Safety, deserve neither Liberty nor Safety." -- B. Franklin, 1759
PGP fingerprint: C4A8 9474 5C4C ADE3 2B8F 656D 47D8 9B65 6951 3011
James Antill
2008-01-08 06:50:59 UTC
Permalink
I've been trying to stay out of this flamewa^W thread, but...
Post by Nils Philippsen
Post by Enrico Scholz
But python or other bloaty scripting languages are not a solution and
completely unacceptable at this place.
"Bloaty" is something that could be solved, don't you think? If python
was split up in a base package that contained the directory structure,
the binaries, the (small) documentation and only basic modules (with no
additional requirements) and a package containing the rest of the
modules, much of the (dep-)bloat wouldn't be an issue if init scripts
(ehh modules) would be limited to that basic set. With the set of
modules I would choose, such a python-base package would have less
dependencies than bash and only account for roughly 2.6MB on the disk as
well -- bash takes up about 5.1MB.
From a Fed-9 python build:

% rpm --qf '%{name} %{size}\n' -q python python-libs python-docs \
python-test python-devel python-tools
python 17070643
python-libs 1505949
python-docs 17804516
python-test 11902909
python-devel 3124644
python-tools 2793600

...all of which come from the python tarball, and only the top two of
which you actually need to run scripts[1].
I looked at getting python itself down a bit more, but it would be a
lot of work getting the deps. correct for "python-minimal" (or whatever)
and even worse is that a lot of things that you'd think of as "non-core"
are required by core applications like yum or setroubleshoot.


[1] tkinter removed to protect any innocents who may be reading this :).
--
James Antill -- <james.antill at redhat.com>
"Please, no. Let's not pull in a dependency for something as simple as a
string library." -- Kristian H?gsberg <krh at redhat.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080108/78331b93/attachment.bin
Lennart Poettering
2008-01-07 15:07:38 UTC
Permalink
Post by Enrico Scholz
Post by Casey Dahlin
Post by Kevin Kofler
A shell which emulates POSIX process handling in-process and uses
direct builtin function calls for commands like sed [...] Pipes and
the like would also have to be emulated
For what do you need 'sed' or pipes to start/stop a daemon?
It appears at least 13 times in our current init system.
I think, nobody doubts that current initsystem is the worst one of
the major linux distributions. By changing paradigm from forking to
non-forking daemon you can avoid all the complicated 'stop' code;
e.g. 'start' will be
| pid = fork();
| if (pid==0) { /* ... */ execve(...); }
and 'stop' be
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
This is not as simple as it might appear. Well behaving daemons don't
detach before initialization is complete. This fact is implicitly used
by SysV init to make sure that daemons which depend on each other
(like in "Avahi needs D-Bus") to start in order, one after the
other, but only after the predependencies have started up completely.

There have been several ideas how signalling of startup completion
could be handled in a better way. Some suggest that waiting for D-Bus
name to appear might be a good solution. Scott Remnant had the idea to
run "raise(SIGSTOP)" in the daemon after initialization
completed. Then, the parent process will be signalled via SIGCHLD and
as soon as it is it will kill(child, SIGCONT) and know that the next
daemon can be started.

Lennart
--
Lennart Poettering Red Hat, Inc.
lennart [at] poettering [dot] net ICQ# 11060553
http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Enrico Scholz
2008-01-07 16:31:29 UTC
Permalink
Post by Lennart Poettering
Post by Enrico Scholz
the major linux distributions. By changing paradigm from forking to
non-forking daemon you can avoid all the complicated 'stop' code;
e.g. 'start' will be
| pid = fork();
| if (pid==0) { /* ... */ execve(...); }
and 'stop' be
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
This is not as simple as it might appear. Well behaving daemons don't
detach before initialization is complete.
That has the problem that heuristics like pid-files must be used to
check/modify status of daemons.

Has the problem too that only one state of "complete" is possible. But
e.g. for dhclient the state 'link was brought up' might be interesting.
Post by Lennart Poettering
This fact is implicitly used by SysV init to make sure that daemons
which depend on each other (like in "Avahi needs D-Bus")
meaning of "complete" depends on the program. E.g. waiting for
completeness of udevd can be done by 'wait-for /dev/.udev'. It is an
implementation detail of the initsystem, whether to detect it with
external tools or by adding builtin functionality. E.g. udevd init
might like like


--- system/udevd ---
need = scripts/udevd-wait

--- scripts/udevd-wait
need = daemon/udevd-daemon
exec = some-inotify-tool --wait-for-file /dev/.udev # external tool
wait_for = /dev/.udev # builtin

--- daemon/udevd-daemon
daemon = udevd




Enrico
Till Maas
2008-01-08 18:01:43 UTC
Permalink
Post by Enrico Scholz
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
But python or other bloaty scripting languages are not a solution and
completely unacceptable at this place.
Imho there is some code missing, that the pid really belongs to the service,
e.g. when the service died/crashed and the pid file still exists, a wrong
process can be killed here. Also I guess it would be better to first try to
SIGTERM the all services that should be terminated, then wait, and then send
the SIGKILL instead of waiting for each process independently.

Regards,
Till
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: This is a digitally signed message part.
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080108/3c7029cd/attachment.bin
Enrico Scholz
2008-01-08 18:48:29 UTC
Permalink
Post by Till Maas
Post by Enrico Scholz
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
Imho there is some code missing, that the pid really belongs to the service,
no; as already written, modern initsystems use non-forking daemons where
such checks are not needed anymore.
Post by Till Maas
e.g. when the service died/crashed and the pid file still exists
pidfiles are ancient hacks not required by modern initsystems anymore.
Post by Till Maas
, a wrong process can be killed here. Also I guess it would be better
to first try to SIGTERM the all services that should be terminated,
then wait, and then send the SIGKILL instead of waiting for each
process independently.
stop sequence happens in reverted order and can be done in parallel, but
there must by synchronization points (e.g. 'httpd' + 'ftpd' -> sync ->
'udhcpc' -> sync -> 'udevd')



Enrico
Casey Dahlin
2008-01-08 18:57:13 UTC
Permalink
Post by Enrico Scholz
Post by Till Maas
Post by Enrico Scholz
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
Imho there is some code missing, that the pid really belongs to the service,
no; as already written, modern initsystems use non-forking daemons where
such checks are not needed anymore.
It is up to the designer of the app whether it forks or not, and while
there may be an argument that one way is better or worse, the init
maintainers cannot guarantee the behavior of that which their system
must start.
Post by Enrico Scholz
Post by Till Maas
e.g. when the service died/crashed and the pid file still exists
pidfiles are ancient hacks not required by modern initsystems anymore.
Post by Till Maas
, a wrong process can be killed here. Also I guess it would be better
to first try to SIGTERM the all services that should be terminated,
then wait, and then send the SIGKILL instead of waiting for each
process independently.
stop sequence happens in reverted order and can be done in parallel, but
there must by synchronization points (e.g. 'httpd' + 'ftpd' -> sync ->
'udhcpc' -> sync -> 'udevd')
Enrico
Enrico Scholz
2008-01-08 23:07:29 UTC
Permalink
Post by Casey Dahlin
Post by Enrico Scholz
no; as already written, modern initsystems use non-forking daemons
where such checks are not needed anymore.
It is up to the designer of the app whether it forks or not, and while
there may be an argument that one way is better or worse, the init
maintainers cannot guarantee the behavior of that which their system
must start.
1. most (yes, I have numbers) daemons have some kind of --do-not-fork
switch

2. most (sorry, I do not have numbers there) daemons have some kind of
bugzilla where you can report patches and enhancements



Enrico
James Antill
2008-01-08 20:38:10 UTC
Permalink
Post by Enrico Scholz
Post by Till Maas
Post by Enrico Scholz
| kill(pid, SIGTERM); /* wait for timeout/sigchld */ kill(pid, SIGKILL);
Imho there is some code missing, that the pid really belongs to the service,
no; as already written, modern initsystems use non-forking daemons where
such checks are not needed anymore.
Post by Till Maas
e.g. when the service died/crashed and the pid file still exists
pidfiles are ancient hacks not required by modern initsystems anymore.
That's just not true, for example:

http://www.initng.org/browser/initng-ifiles/trunk/initfiles/daemon/exim/listener.ii

...maybe you mean that they support it, and _in your opinion_
application code should change to use that model ... but that's a very
different thing.
--
James Antill <james.antill at redhat.com>
Red Hat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20080108/51d8ab0a/attachment.bin
Enrico Scholz
2008-01-08 23:29:51 UTC
Permalink
Post by James Antill
Post by Enrico Scholz
no; as already written, modern initsystems use non-forking daemons
where such checks are not needed anymore.
...
pidfiles are ancient hacks not required by modern initsystems anymore.
http://www.initng.org/browser/initng-ifiles/trunk/initfiles/daemon/exim/listener.ii
initng has code to deal with old daemons which do not have a --do-not-fork
option. E.g. they *can* look for a pidfile after some time and inject back
the pid.



Enrico
Alan Cox
2008-01-07 12:03:49 UTC
Permalink
Post by Kevin Kofler
AFAIK, busybox still forks whereever a regular POSIX shell forks, so if the
amount of forks is the problem, AFAICT busybox will resolve absolutely nothing.
Fork should be pretty cheap - although that depends how much memory is unshared
by each of the resulting tasks. A smaller cleaner shell such as rc (which was
designed for this job in plan 9) or ash might well perform better. I'm dubious
it would be a big difference but someone can bench it.

Alan
Les Mikesell
2008-01-07 16:35:43 UTC
Permalink
Post by Alan Cox
Post by Kevin Kofler
AFAIK, busybox still forks whereever a regular POSIX shell forks, so if the
amount of forks is the problem, AFAICT busybox will resolve absolutely nothing.
Fork should be pretty cheap - although that depends how much memory is unshared
by each of the resulting tasks. A smaller cleaner shell such as rc (which was
designed for this job in plan 9) or ash might well perform better. I'm dubious
it would be a big difference but someone can bench it.
If a unix-like system can't fork/exec at a rate suitable to handle
starting it's initial processes you should throw the whole system out
and start over, because it will be useless even after you get it
running. I think the real problem you need to solve is the number of
file opens that happen between boot up and the end of the init script
processing. This: http://kernelslacker.livejournal.com/35270.html
and the presentation on the topic at Oscon looked pretty horrible and
won't be fixed by using a dumber shell to parse the scripts. And note
that the suggestion to break out lines of configurations into individual
files for easier programmatic editing just compounds this already
serious problem.
--
Les Mikesell
lesmikesell at gmail.com
Casey Dahlin
2008-01-07 17:38:02 UTC
Permalink
Post by Les Mikesell
Post by Alan Cox
Post by Kevin Kofler
AFAIK, busybox still forks whereever a regular POSIX shell forks, so
if the amount of forks is the problem, AFAICT busybox will resolve
absolutely nothing.
Fork should be pretty cheap - although that depends how much memory is unshared
by each of the resulting tasks. A smaller cleaner shell such as rc (which was
designed for this job in plan 9) or ash might well perform better. I'm dubious
it would be a big difference but someone can bench it.
If a unix-like system can't fork/exec at a rate suitable to handle
starting it's initial processes you should throw the whole system out
and start over, because it will be useless even after you get it
running. I think the real problem you need to solve is the number of
file opens that happen between boot up and the end of the init script
processing. This: http://kernelslacker.livejournal.com/35270.html
and the presentation on the topic at Oscon looked pretty horrible and
won't be fixed by using a dumber shell to parse the scripts. And note
that the suggestion to break out lines of configurations into
individual files for easier programmatic editing just compounds this
already serious problem.
Its not the fork or exec per se. It is the disk IO associated with
loading the binary images. Normally this isn't too much of an issue, but
in the highly IO-sensitive init process it can cause huge issues.
Remember, seek time is the big issue with disk IO, so size of data to
load is not the metric to go by.
nodata
2008-01-07 17:46:29 UTC
Permalink
Post by Casey Dahlin
Post by Les Mikesell
Post by Alan Cox
Post by Kevin Kofler
AFAIK, busybox still forks whereever a regular POSIX shell forks, so
if the amount of forks is the problem, AFAICT busybox will resolve
absolutely nothing.
Fork should be pretty cheap - although that depends how much memory is unshared
by each of the resulting tasks. A smaller cleaner shell such as rc (which was
designed for this job in plan 9) or ash might well perform better. I'm dubious
it would be a big difference but someone can bench it.
If a unix-like system can't fork/exec at a rate suitable to handle
starting it's initial processes you should throw the whole system out
and start over, because it will be useless even after you get it
running. I think the real problem you need to solve is the number of
file opens that happen between boot up and the end of the init script
processing. This: http://kernelslacker.livejournal.com/35270.html
and the presentation on the topic at Oscon looked pretty horrible and
won't be fixed by using a dumber shell to parse the scripts. And note
that the suggestion to break out lines of configurations into
individual files for easier programmatic editing just compounds this
already serious problem.
Its not the fork or exec per se. It is the disk IO associated with
loading the binary images. Normally this isn't too much of an issue, but
in the highly IO-sensitive init process it can cause huge issues.
Remember, seek time is the big issue with disk IO, so size of data to
load is not the metric to go by.
So is this frequently loaded binary not being loaded from the disk
cache?
Bill Nottingham
2008-01-07 19:29:02 UTC
Permalink
Its not the fork or exec per se. It is the disk IO associated with loading
the binary images. Normally this isn't too much of an issue, but in the
highly IO-sensitive init process it can cause huge issues. Remember, seek
time is the big issue with disk IO, so size of data to load is not the
metric to go by.
So, switch to a storage metaphor that doesn't penalize seeks.

While that's somewhat of a joke, you do want to avoid over-optimizing for what
may not be the common case.

Bill
Bill Nottingham
2008-01-07 19:27:27 UTC
Permalink
Post by Alan Cox
Fork should be pretty cheap - although that depends how much memory is unshared
by each of the resulting tasks. A smaller cleaner shell such as rc (which was
designed for this job in plan 9) or ash might well perform better. I'm dubious
it would be a big difference but someone can bench it.
ash has been benchmarked. Required rooting out some bashisms from the scripts
(or just calling those specific scripts with bash), but in any case, it didn't
make much difference.

Bill
Ralf Corsepius
2008-01-08 06:29:11 UTC
Permalink
Post by Bill Nottingham
Post by Alan Cox
Fork should be pretty cheap - although that depends how much memory is unshared
by each of the resulting tasks. A smaller cleaner shell such as rc (which was
designed for this job in plan 9) or ash might well perform better. I'm dubious
it would be a big difference but someone can bench it.
ash has been benchmarked. Required rooting out some bashisms from the scripts
(or just calling those specific scripts with bash),
Right - Doing so (== bug fixing) is way over due.
Post by Bill Nottingham
but in any case, it didn't
make much difference.
Well, depends on how things are being implemented.

Some OSes wanted to avoid "bash's bloat" by using ash as /bin/sh - Had
shown not to be a clever idea.

Ralf
nodata
2008-01-06 17:18:37 UTC
Permalink
Post by Tanguy Eric
I just find this
http://www.pardus.org.tr/eng/projeler/comar/SpeedingUpLinuxWithPardus.html maybe it's known but someone could comment this because it seems to gain init time.
Eric
This looks good, because it solves the actual problem that people are
complaining about (unoptimal boot). There are other problems that people
have made clear as part of the unoptimal boot discussion, but as a first
step, maximising i/o usage seems like a good one :)
Continue reading on narkive:
Loading...