Discussion:
raising warning flag on firewalld-default feature
(too old to reply)
Matthew Miller
2012-11-09 14:33:08 UTC
Permalink
https://fedoraproject.org/wiki/Features/firewalld-default

We have an accepted feature for Firewalld to be the default in Fedora 18.

The old scripts are primitive and can't handle dynamic environments very
well, so having something new and modern is admirable. The lokkit family of
GUI config tools is primative enough to be considered dangerous. And a lot
of integration work has been done in NetworkManager, libvirt, and a bunch of
other places.

But, I think we should strongly consider pushing this to F19, because:

- this turns out to be a big change!
- there's little to no documentation
- the UI is very confusing, with a large number of "zones" and no apparent
way to configure those zones
- toolset is not yet robust -- has funny things like `firewall-cmd
--enable` enables *panic mode*.
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources

The alternative is to enable it by default in some cases but not in others,
but I think that's just confusing. We should wait until it's ready and then
turn it on everywhere.

I think this bug is illustrative of the problems we're going to see if we
ship as-is: <https://bugzilla.redhat.com/show_bug.cgi?id=869625>. Stef isn't
trying to anything crazy, but is both foiled by the lack of options and
confused by the choices that are there. We're going to get a lot more bugs
like this, and worse, unhappy users.

The lack of documentation is really the showstopper here. If we had really
good 1) hand-holding documentation and 2) technical documentation for
admins, I'd be more willing to take the risk. (In an even more ideal world,
the UI would be so well designed that the hand-holding documentation
wouldn't be necessary.)
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Eric H. Christensen
2012-11-09 16:24:30 UTC
Permalink
Post by Matthew Miller
https://fedoraproject.org/wiki/Features/firewalld-default
We have an accepted feature for Firewalld to be the default in Fedora 18.
This replaces iptables and ip6tables? Perhaps I have had my head in the sand (I certainly haven't been looking around) but this is the first I've heard of a replacement for iptables. Has firewalld been tested as well as iptables has (which seems to be a fairly bullet-proof solution)?
...
Post by Matthew Miller
- there's little to no documentation
I'd happily help document it in the Fedora Security Guide if I could get the proper content or access to the developers. Heck, I'll even help write stand-alone documentation for this project if needed.
Post by Matthew Miller
The lack of documentation is really the showstopper here. If we had really
good 1) hand-holding documentation and 2) technical documentation for
admins, I'd be more willing to take the risk. (In an even more ideal world,
the UI would be so well designed that the hand-holding documentation
wouldn't be necessary.)
+1

- -Eric "Sparks"
Thomas Woerner
2012-11-09 16:45:23 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Post by Matthew Miller
https://fedoraproject.org/wiki/Features/firewalld-default
We have an accepted feature for Firewalld to be the default in Fedora 18.
This replaces iptables and ip6tables? Perhaps I have had my head in the sand (I certainly haven't been looking around) but this is the first I've heard of a replacement for iptables. Has firewalld been tested as well as iptables has (which seems to be a fairly bullet-proof solution)?
Please have a look at the feature list for F-18.

firewalld replaces system-config-firewall/lokkit, and the iptables and
ip6tables services, not the iptables package and command.

The ip*tables services and also system-config-firewall/lokkit are still
available and also usable after deactivation of the firewalld serice.
With the latest request to move the services of iptables and ip6tables
in a sub package, I will add a requirement to system-config-firewall for
this.
...
Post by Matthew Miller
- there's little to no documentation
I'd happily help document it in the Fedora Security Guide if I could get the proper content or access to the developers. Heck, I'll even help write stand-alone documentation for this project if needed.
I will provide content/help for this.
Post by Matthew Miller
The lack of documentation is really the showstopper here. If we had really
good 1) hand-holding documentation and 2) technical documentation for
admins, I'd be more willing to take the risk. (In an even more ideal world,
the UI would be so well designed that the hand-holding documentation
wouldn't be necessary.)
+1
- -Eric "Sparks"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iQIcBAEBAgAGBQJQnS4+AAoJEIB2q94CS7PRdGcP/1z5O5kgvHDX04E/6t3xhdWv
w2JtwDC3zYc0KlASa+XFPlqFmvQUBngI7Esy3kJ8+mas+bFwVOftTRQhZz13mmfg
C+eKe/rHtL2hEF/EDkWe23FSASrHdK6FNyotK7xxdfh3QYPGmavmFSvlETg6qUdS
kkWRrTCtkro4EirO7KGbW7DDeuzcxqK6IHy6JStdevouwaTqJ/TtdCI2vYJKDTyg
GkxQQwk00GCk7xox5dJq1jdpniVfpQ/pKAVb9BTuQYCaMCuqdv64xg6ggbkXi28T
cIFkdKxNCBw0L5Ecwg3/d4y2OlTAJmBULsAQZ7piFKXFbHPb9CofxCypGSTn5cMw
F9wnr/0geTw3UOxfi0OGNm2Wf0x2B9n7iyYZODxvihdoeg8OGbusPJr9viRYI7tA
47+/95ywXBTcAPxLwSCb3vXG2FImgnzwnaG/9xpKZk4dKAZcxQBxqlgDtBbilv8X
zvr9ArmCG9hdEAojD66AKM5Qmzse+tPaAiDFecGBvlSN3/J2AOrTF9U64Akbkzg9
+uXkV3rk/DhP0JTLXvb8Aizbb9Y51PGO/G7KZH3tCYieaCQbkNdddNbIg3WI4kV1
AEGvDd30vDdAkl17UcguV6iwPwCP0tFs9GNcJRCEninL+/bQmVs2PcpYJ5+oPBsi
vUc791SABXCttPkm1X/A
=E0LH
-----END PGP SIGNATURE-----
Eric H. Christensen
2012-11-09 16:52:56 UTC
Permalink
Post by Thomas Woerner
Post by Eric H. Christensen
I'd happily help document it in the Fedora Security Guide if I could get the proper content or access to the developers. Heck, I'll even help write stand-alone documentation for this project if needed.
I will provide content/help for this.
Awesome, thanks! Hit me up on IRC (Sparks on freenode) or email when you get some time.

- -Eric
Reindl Harald
2012-11-09 18:45:10 UTC
Permalink
Post by Thomas Woerner
Please have a look at the feature list for F-18.
firewalld replaces system-config-firewall/lokkit, and the iptables and ip6tables services, not the iptables package
and command.
The ip*tables services and also system-config-firewall/lokkit are still available and also usable after
deactivation of the firewalld serice. With the latest request to move the services of iptables and ip6tables in a
sub package, I will add a requirement to system-config-firewall for this
PLEASE do not "Require: system-config-firewall"
this would pull useless dependencies

what we (users) really need is "iptables.service" as it was and
working "/sbin/iptables-save > /etc/sysconfig/iptables" to lod
the with whatever shell script generated "/etc/sysconfig/iptables"
so satisfy over many years perfect working setups for

(the same for iptables6.service)

* firewalls
* NAT
* routing

as example i have a large shellscript
with the following start

$IPTABLES -P INPUT DROP
$IPTABLES -P FORWARD DROP
$IPTABLES -F
$IPTABLES -X
CHAINS=`cat /proc/net/ip_tables_names 2>/dev/null`
for i in $CHAINS; do $IPTABLES -t $i -F; done && echo "Flush OK" || echo "Flush FAILED"
for i in $CHAINS; do $IPTABLES -t $i -X; done && echo "Clear OK" || echo "Clear FAILED"
for i in $CHAINS; do $IPTABLES -t $i -Z; done

and ending with "/sbin/iptables-save > /etc/sysconfig/iptables"
after that any needed rules are added with iptables-command

this script is distributed to a LOT of machines of any type

at the begin it has basic rules for any machine (accept, block, reject)
followed by a lot of

if [ "$HOSTNAME" == "hostname" ]; then
<specific rules>
fi

this is maintained on a staging server, distributed to any amchine
and called with "ssh root at host '/scirpts/iptables.sh"

for other networks / routers / nat-gateways outside the main network
a fork of this thing exists, using over years grown knowledge and
adds specific rules, mostly controlled by a lot of variables at the
begin

call this script does NOt interrupt connections
it handles really a lot of specific filters
it works like a charme

these setups does not need firewalld at all nor do
they need any dependency of GUI/TUI tools





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 259 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121109/fe58a361/attachment.sig>
Thomas Woerner
2012-11-12 09:25:53 UTC
Permalink
Post by Reindl Harald
Post by Thomas Woerner
Please have a look at the feature list for F-18.
firewalld replaces system-config-firewall/lokkit, and the iptables and ip6tables services, not the iptables package
and command.
The ip*tables services and also system-config-firewall/lokkit are still available and also usable after
deactivation of the firewalld serice. With the latest request to move the services of iptables and ip6tables in a
sub package, I will add a requirement to system-config-firewall for this
PLEASE do not "Require: system-config-firewall"
this would pull useless dependencies
What I meant: Add a requirement for iptables-services to
system-config-firewall-base, this is currently not there.
Post by Reindl Harald
what we (users) really need is "iptables.service" as it was and
working "/sbin/iptables-save > /etc/sysconfig/iptables" to lod
the with whatever shell script generated "/etc/sysconfig/iptables"
so satisfy over many years perfect working setups for
(the same for iptables6.service)
* firewalls
* NAT
* routing
as example i have a large shellscript
with the following start
$IPTABLES -P INPUT DROP
$IPTABLES -P FORWARD DROP
$IPTABLES -F
$IPTABLES -X
CHAINS=`cat /proc/net/ip_tables_names 2>/dev/null`
for i in $CHAINS; do $IPTABLES -t $i -F; done && echo "Flush OK" || echo "Flush FAILED"
for i in $CHAINS; do $IPTABLES -t $i -X; done && echo "Clear OK" || echo "Clear FAILED"
for i in $CHAINS; do $IPTABLES -t $i -Z; done
and ending with "/sbin/iptables-save > /etc/sysconfig/iptables"
after that any needed rules are added with iptables-command
this script is distributed to a LOT of machines of any type
at the begin it has basic rules for any machine (accept, block, reject)
followed by a lot of
if [ "$HOSTNAME" == "hostname" ]; then
<specific rules>
fi
this is maintained on a staging server, distributed to any amchine
and called with "ssh root at host '/scirpts/iptables.sh"
for other networks / routers / nat-gateways outside the main network
a fork of this thing exists, using over years grown knowledge and
adds specific rules, mostly controlled by a lot of variables at the
begin
call this script does NOt interrupt connections
it handles really a lot of specific filters
it works like a charme
these setups does not need firewalld at all nor do
they need any dependency of GUI/TUI tools
Yes, full ack.

You will be able to use it after switching off firewalld.service and
enabling iptables.service and ip6tables.service.

I will add a script for switching from and to dynamic/static mode:
switch-firewall
Thomas Woerner
2012-11-09 16:32:14 UTC
Permalink
Post by Matthew Miller
https://fedoraproject.org/wiki/Features/firewalld-default
We have an accepted feature for Firewalld to be the default in Fedora 18.
The old scripts are primitive and can't handle dynamic environments very
well, so having something new and modern is admirable. The lokkit family of
GUI config tools is primative enough to be considered dangerous. And a lot
of integration work has been done in NetworkManager, libvirt, and a bunch of
other places.
- this turns out to be a big change!
- there's little to no documentation
Have you had a look at the man pages?
Post by Matthew Miller
- the UI is very confusing, with a large number of "zones" and no apparent
way to configure those zones
Go to the persistent view and you can configure zones, services and
icmptypes.
Post by Matthew Miller
- toolset is not yet robust -- has funny things like `firewall-cmd
--enable` enables *panic mode*.
Nice find. You are the first to get this. Will work on it.
Post by Matthew Miller
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources
You can use the old firewall environment for static firewall use cases.
Everything is still there.

Firewalld is using about 12M of memory (RES), produces only a small
amount of wakeups (< 0.1) if idle. Where is the non-trivial use of
system resources.
Post by Matthew Miller
The alternative is to enable it by default in some cases but not in others,
but I think that's just confusing. We should wait until it's ready and then
turn it on everywhere.
I think this bug is illustrative of the problems we're going to see if we
ship as-is: <https://bugzilla.redhat.com/show_bug.cgi?id=869625>. Stef isn't
trying to anything crazy, but is both foiled by the lack of options and
confused by the choices that are there. We're going to get a lot more bugs
like this, and worse, unhappy users.
libvirt is creating the firewall rules for guests - it is doing this
with the old static model, where you loose these rules in case of other
firewall changes, or with firewalld, but here changes are dynamic.
Post by Matthew Miller
The lack of documentation is really the showstopper here. If we had really
good 1) hand-holding documentation and 2) technical documentation for
admins, I'd be more willing to take the risk. (In an even more ideal world,
the UI would be so well designed that the hand-holding documentation
wouldn't be necessary.)
Matthew Miller
2012-11-09 16:57:12 UTC
Permalink
Post by Thomas Woerner
Post by Matthew Miller
- this turns out to be a big change!
- there's little to no documentation
Have you had a look at the man pages?
I missed the top-level man page and was looking at firewall-cmd, which is
not very helpful on its own. Starting from firewalld is much more helpful.
(Thanks!)

The Zone man page dumps me right into reading XML. :) This is the technical
documentation I was referring to, and I'm glad to see it _is_ there -- sorry
I missed it. I'm still not clear on some concepts, though -- particularly,
a zone is described as defining the "trust level of the interface used for a
connection", but in the man page for zones, "trust" isn't mentioned at all
-- instead, they appear to be the config files for firewall chains.

But I can get into my specific confusion in a separate thread. For the point
of view of the feature, we need to get some of this into web pages and maybe
online help for the GUI applet.
Post by Thomas Woerner
Post by Matthew Miller
- the UI is very confusing, with a large number of "zones" and no apparent
way to configure those zones
Go to the persistent view and you can configure zones, services and
icmptypes.
I can certainly check and uncheck services and other things within zones,
but the GUI gives me no idea about what the zones mean and neither a way to
learn that nor a way to tell it -- I'd expect at least _one_ of those. I see
there's a "work" zone -- how does firewalld know I'm on the work network and
not at home or at a coffee shop?
Post by Thomas Woerner
Post by Matthew Miller
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources
You can use the old firewall environment for static firewall use
cases. Everything is still there.
Can I use them *both together*? If so, okay. If not, we should keep entirely
with the old one until this is really ready to take over.
Post by Thomas Woerner
Firewalld is using about 12M of memory (RES), produces only a small
amount of wakeups (< 0.1) if idle. Where is the non-trivial use of
system resources.
That. That right there. When the net result of that is _no work done ever_,
multipled by a thousand of million, it's really not a good use of the
world's resources.

Even on a dynamic system, it's going to be idle most of the time, right?
Couldn't this be entirely D-BUS activated and exit after making changes?
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Matthew Miller
2012-11-13 15:02:26 UTC
Permalink
Post by Matthew Miller
Post by Thomas Woerner
Post by Matthew Miller
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources
You can use the old firewall environment for static firewall use
cases. Everything is still there.
Can I use them *both together*? If so, okay. If not, we should keep entirely
with the old one until this is really ready to take over.
This is still unclear to me. Anaconda is pulling in firewalld for
post-install configuration. Do we still _really_ have the option of the old
firewall?
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Thomas Woerner
2012-11-13 16:31:25 UTC
Permalink
Post by Matthew Miller
Post by Matthew Miller
Post by Thomas Woerner
Post by Matthew Miller
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources
You can use the old firewall environment for static firewall use
cases. Everything is still there.
Can I use them *both together*? If so, okay. If not, we should keep entirely
with the old one until this is really ready to take over.
This is still unclear to me. Anaconda is pulling in firewalld for
post-install configuration. Do we still _really_ have the option of the old
firewall?
You can not use both at the same time, as they are doing things
completely different. The static firewall stuff is not active by
default, but everything is available.
Matthew Miller
2012-11-13 15:03:40 UTC
Permalink
Post by Matthew Miller
Post by Thomas Woerner
Post by Matthew Miller
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources
You can use the old firewall environment for static firewall use
cases. Everything is still there.
Can I use them *both together*? If so, okay. If not, we should keep entirely
with the old one until this is really ready to take over.
This is still unclear to me. Anaconda is pulling in firewalld for
post-install configuration. Do we still _really_ have the option of the old
firewall?
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Adam Williamson
2012-11-13 17:52:32 UTC
Permalink
Post by Matthew Miller
Post by Matthew Miller
Post by Thomas Woerner
Post by Matthew Miller
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources
You can use the old firewall environment for static firewall use
cases. Everything is still there.
Can I use them *both together*? If so, okay. If not, we should keep entirely
with the old one until this is really ready to take over.
This is still unclear to me. Anaconda is pulling in firewalld for
post-install configuration. Do we still _really_ have the option of the old
firewall?
We can in fact stop pulling in firewalld for post-install configuration
in most cases, I think, I'm talking to twoerner/anaconda team about
that. You can certainly remove it post-install and go back to using
iptables / s-c-f, all the packages and services still exist, there is no
dependency problem, and it still works, I've tested that. Actually, if
you have an F18 system you 'yum update'd from F17 you already probably
have this config, as there is no magic to do the switch to firewalld on
upgrade, you just carry on with iptables. That's what I have here.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
Richard W.M. Jones
2012-11-10 09:23:21 UTC
Permalink
Post by Matthew Miller
- this turns out to be a big change!
- there's little to no documentation
- the UI is very confusing, with a large number of "zones" and no apparent
way to configure those zones
- toolset is not yet robust -- has funny things like `firewall-cmd
--enable` enables *panic mode*.
- no way to run once and exit for cloud guests with *non-dynamic* firewall
needs, and it's a non-trivial user of system resources
- depends on Python stack

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
Kevin Kofler
2012-11-10 20:53:13 UTC
Permalink
Post by Richard W.M. Jones
- depends on Python stack
+1, we really need to get Python out of the minimal installation.

The focus should be on replacing the existing Python-based packages in the
minimum set (e.g. yum) by native replacements (e.g. zif). Adding more Python
stuff with additional Python dependencies is a step backwards.

I really don't understand why a core system component such as firewalld is
implemented in Python!

Kevin Kofler
Seth Vidal
2012-11-12 06:28:10 UTC
Permalink
Post by Kevin Kofler
Post by Richard W.M. Jones
- depends on Python stack
+1, we really need to get Python out of the minimal installation.
The focus should be on replacing the existing Python-based packages in the
minimum set (e.g. yum) by native replacements (e.g. zif). Adding more Python
stuff with additional Python dependencies is a step backwards.
I really don't understand why a core system component such as firewalld is
implemented in Python!
Yum will likely be replaced with dnf afaik. I don't think zif is under
consideration at all.

-sv
Jan Zelený
2012-11-12 07:37:25 UTC
Permalink
Post by Seth Vidal
Post by Kevin Kofler
Post by Richard W.M. Jones
- depends on Python stack
+1, we really need to get Python out of the minimal installation.
The focus should be on replacing the existing Python-based packages in the
minimum set (e.g. yum) by native replacements (e.g. zif). Adding more
Python stuff with additional Python dependencies is a step backwards.
I really don't understand why a core system component such as firewalld is
implemented in Python!
Yum will likely be replaced with dnf afaik. I don't think zif is under
consideration at all.
Yes, that's the plan. But dnf is still Python. So if we really want to get
Python out of minimal install, there is a room for possible alternatives I
guess. But none of this is certainly happening for at least a year or two, we
don't want to break things by rushing.

Thanks
Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121112/5e714b4f/attachment.sig>
Kevin Kofler
2012-11-12 18:54:05 UTC
Permalink
Post by Jan Zelený
Yes, that's the plan. But dnf is still Python. So if we really want to get
Python out of minimal install, there is a room for possible alternatives I
guess.
Right. We need to stop writing core system components in scripting
languages!
Post by Jan Zelený
But none of this is certainly happening for at least a year or two, we
don't want to break things by rushing.
But that doesn't justify adding MORE Python stuff to the minimal install.

Kevin Kofler
Miloslav Trmač
2012-11-12 19:15:48 UTC
Permalink
Post by Kevin Kofler
Post by Jan Zelený
Yes, that's the plan. But dnf is still Python. So if we really want to get
Python out of minimal install, there is a room for possible alternatives I
guess.
Right. We need to stop writing core system components in scripting
languages!
Well, there _are_ significant advantages to using a higer-level
language than C.[1] Using one of the higher-level languages as a
primary development language on par with C often increases the quality
of the software and the time to develop it, in return for an
acceptable loss of speed. If firewalld were an on-demand,
automatically-terminating D-Bus service, I can't really see any reason
to object to a using higher-level language.

Unfortunately, there's seems to be something significantly
wrong/disliked about every candidate higher-level language. For
better or worse, the Fedora world is mostly standardized on using
Python in such situations. Is it perfect? No. Is it better than
nothing? I think so.
Mirek

[1] At least two to mention:
- Automatic memory management, with all associated lack of bugs and
simplified code
- Much easier access to higher-level (and more efficient!) data
structures. C programs frequently use linked lists instead of e.g.
hash tables simply because maintaining hash tables and the associated
memory allocation is just too complex.
Chris Adams
2012-11-12 19:25:29 UTC
Permalink
Post by Miloslav Trmač
- Much easier access to higher-level (and more efficient!) data
structures. C programs frequently use linked lists instead of e.g.
hash tables simply because maintaining hash tables and the associated
memory allocation is just too complex.
Hash table management was a part of my computer science core curriculum;
it isn't _that_ hard.

Also, for C programmers, there's standard library hash management
functions that are easy to use (never try to re-invent the wheel!). See
"man hcreate_r" (although it is unfortunately just a GNU extension to
have multiple hash tables).

Now, given that, I still write most of my system admin stuff in perl.
--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
Miloslav Trmač
2012-11-12 19:36:11 UTC
Permalink
Post by Chris Adams
Post by Miloslav Trmač
- Much easier access to higher-level (and more efficient!) data
structures. C programs frequently use linked lists instead of e.g.
hash tables simply because maintaining hash tables and the associated
memory allocation is just too complex.
Hash table management was a part of my computer science core curriculum;
it isn't _that_ hard.
Sure. Now create a hash table indexed by a tuple, or for more
challenge by an unordered set of strings. Or have all items of the
same type hashed by three different fields for fast lookups.

Can it be done? Absolutely.
Will it be bug-free? Probably not on the first try.
Do C programmers actually write such code? So far as I have seen,
usually not; if something simpler and less efficient works for version
1, it stays that way until it is becomes a very noticeable bottleneck.
Mirek
Chris Adams
2012-11-12 19:43:54 UTC
Permalink
Post by Miloslav Trmač
Sure. Now create a hash table indexed by a tuple, or for more
challenge by an unordered set of strings. Or have all items of the
same type hashed by three different fields for fast lookups.
If I were trying to do all that, I'd probably just go with an in-memory
SQLite table.
--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
Steve Grubb
2012-11-12 19:34:12 UTC
Permalink
Post by Kevin Kofler
Post by Jan Zelený
Yes, that's the plan. But dnf is still Python. So if we really want to
get Python out of minimal install, there is a room for possible
alternatives I guess.
Right. We need to stop writing core system components in scripting
languages!
Well, there are significant advantages to using a higer-level
language than C.[1]
But the problem I see is a lot of libraries are wrapped by swig, which leaks
memory like a sieve. If swig didn't generate such leaky code, Python based
daemons wouldn't be as scary.

-Steve
Alek Paunov
2012-11-15 00:49:18 UTC
Permalink
Post by Steve Grubb
But the problem I see is a lot of libraries are wrapped by swig, which leaks
memory like a sieve. If swig didn't generate such leaky code, Python based
daemons wouldn't be as scary.
IMHO, Python is one of the best ways to express management logic. As
have stated elsewhere in the thread, from the footprint perspective,
python-libs can be decomposed to something including much smaller
python-libs-core. The remaining problem is the quality of part of the C
bindings. Otherwise, it is great advantage for Fedora that huge part of
the system tools are already polished in Python (my own experience is
that the C code in many tools/libs shipped with Fedora is of relative
poor quality, as already Miloslav explained).

David Malcolm already has some real results addressing the Python
bindings issues trough static analysis based on his great
gcc-python-plugin/libcpychecker work. In the future, this machinery can
be used for bug free bindings generation too.

Another option for addressing this is ... to just get rid of part of
them in their current form.

LuaJIT 2.0 is a reimplementation of Lua scripting language, which has
build the reputation of one of the fastest compilers (comparable only
with gcc) in the course of his 3 years! beta cycle (2.0 final release
was announced early this week)

The size of luajit.so (interpreter + trace compiler + C parser for the
FFI definitions) is 452088 bytes (on x86_64, for comparison sqlite3.so,
which is a python requirement is 677672)

Unlike other FFI implementations LuaJIT FFI works compiling the
functions calls/C structs access to native code. Another difference is
that the foreign C definitions are presented in native C syntax:

http://luajit.org/ext_ffi_tutorial.html#zlib

On top of this kind of FFI, LuaJIT already have bunch of library
bindings - as instance the kernel ABI wrapper:

https://github.com/justincormack/ljsyscall

For a long time already Fedora ships lunatic-python (loading and
bridging the Lua interpreter as python module)

More recent and feature complete bridge is:

https://github.com/scoder/lupa

from the author of lxml module.

So, to me it seems natural joining all above together to start thinking
for replacing the classic python C bindings with thin textual or
bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
dependencies like Anaconda (thanks to lupa, Lua objects behaves like
Python ones in Python). The result will be order of magnitude easier for
maintenance (towards the APIs evolution) and faster code.

Furthermore, we can get LuaJIT for free, because it is full API/ABI
compatible with current Fedora (PUC Rio) Lua 5.1, which is pulled by RPM
anyway - we can just replace Lua with LuaJIT as RPM provider for F20.

Kind Regards,
Alek

(*) Unlike original Lua, LuaJIT bytecode is portable across architectures.
M. Edward (Ed) Borasky
2012-11-15 02:51:58 UTC
Permalink
How much Python code are you proposing "someone" ports to Lua? ;-)
Post by Steve Grubb
But the problem I see is a lot of libraries are wrapped by swig, which leaks
memory like a sieve. If swig didn't generate such leaky code, Python based
daemons wouldn't be as scary.
IMHO, Python is one of the best ways to express management logic. As have
stated elsewhere in the thread, from the footprint perspective, python-libs
can be decomposed to something including much smaller python-libs-core. The
remaining problem is the quality of part of the C bindings. Otherwise, it is
great advantage for Fedora that huge part of the system tools are already
polished in Python (my own experience is that the C code in many tools/libs
shipped with Fedora is of relative poor quality, as already Miloslav
explained).
David Malcolm already has some real results addressing the Python bindings
issues trough static analysis based on his great
gcc-python-plugin/libcpychecker work. In the future, this machinery can be
used for bug free bindings generation too.
Another option for addressing this is ... to just get rid of part of them in
their current form.
LuaJIT 2.0 is a reimplementation of Lua scripting language, which has build
the reputation of one of the fastest compilers (comparable only with gcc) in
the course of his 3 years! beta cycle (2.0 final release was announced early
this week)
The size of luajit.so (interpreter + trace compiler + C parser for the FFI
definitions) is 452088 bytes (on x86_64, for comparison sqlite3.so, which is
a python requirement is 677672)
Unlike other FFI implementations LuaJIT FFI works compiling the functions
calls/C structs access to native code. Another difference is that the
http://luajit.org/ext_ffi_tutorial.html#zlib
On top of this kind of FFI, LuaJIT already have bunch of library bindings -
https://github.com/justincormack/ljsyscall
For a long time already Fedora ships lunatic-python (loading and bridging
the Lua interpreter as python module)
https://github.com/scoder/lupa
from the author of lxml module.
So, to me it seems natural joining all above together to start thinking for
replacing the classic python C bindings with thin textual or bytecode(*)
LuaJIT/FFI shims in benefit of things with great set of dependencies like
Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python).
The result will be order of magnitude easier for maintenance (towards the
APIs evolution) and faster code.
Furthermore, we can get LuaJIT for free, because it is full API/ABI
compatible with current Fedora (PUC Rio) Lua 5.1, which is pulled by RPM
anyway - we can just replace Lua with LuaJIT as RPM provider for F20.
Kind Regards,
Alek
(*) Unlike original Lua, LuaJIT bytecode is portable across architectures.
--
devel mailing list
devel at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
--
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?
Alek Paunov
2012-11-15 03:21:20 UTC
Permalink
Post by M. Edward (Ed) Borasky
How much Python code are you proposing "someone" ports to Lua? ;-)
I am proposing mostly porting C code to Lua/FFI (the code of the
problematic Python C bindings)
Florian Weimer
2012-11-15 11:18:45 UTC
Permalink
Post by Alek Paunov
So, to me it seems natural joining all above together to start thinking
for replacing the classic python C bindings with thin textual or
bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
dependencies like Anaconda (thanks to lupa, Lua objects behaves like
Python ones in Python). The result will be order of magnitude easier for
maintenance (towards the APIs evolution) and faster code.
LuaJIT is currently not available for our secondary architectures, so
this would need fixing first. ("not available" meaning exactly that,
there is no fallback to a slow C interpreter mode or something like that.)

It's also not clear how significant the actual memory savings will be
because some of the Lua OO frameworks have quite a bit of overhead. And
the eventually arriving abrt plugin will bring some weight with it, too.

This is nothing against Lua or LuaJIT. I just don't think that
switching programming languages will magically reduce resource
consumption. This has to be an explicit design goal, or else it is not
likely to happen. I'm sure it's possible to write resource-conserving
Python code, too. It's just the baseline overhead from the run-time
system which is unavoidable, but that's fairly small for Python
(apparently less than 2 MB of unshared RSS per process).
--
Florian Weimer / Red Hat Product Security Team
drago01
2012-11-15 13:18:32 UTC
Permalink
Post by Alek Paunov
So, to me it seems natural joining all above together to start thinking
for replacing the classic python C bindings with thin textual or
bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
dependencies like Anaconda (thanks to lupa, Lua objects behaves like
Python ones in Python). The result will be order of magnitude easier for
maintenance (towards the APIs evolution) and faster code.
LuaJIT is currently not available for our secondary architectures, so this
would need fixing first. ("not available" meaning exactly that, there is no
fallback to a slow C interpreter mode or something like that.)
It's also not clear how significant the actual memory savings will be
because some of the Lua OO frameworks have quite a bit of overhead. And the
eventually arriving abrt plugin will bring some weight with it, too.
This is nothing against Lua or LuaJIT. I just don't think that switching
programming languages will magically reduce resource consumption. This has
to be an explicit design goal, or else it is not likely to happen. I'm sure
it's possible to write resource-conserving Python code, too. It's just the
baseline overhead from the run-time system which is unavoidable, but that's
fairly small for Python (apparently less than 2 MB of unshared RSS per
process).
Porting yum to python3 might be an easier task ... from the python 3.3
changelog [1]:

"The storage of Unicode strings now depends on the highest codepoint
in the string:

pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per codepoint;
BMP strings (U+0000-U+FFFF) use 2 bytes per codepoint;
non-BMP strings (U+10000-U+10FFFF) use 4 bytes per codepoint.

The net effect is that for most applications, memory usage of string
storage should decrease significantly - especially compared to former
wide unicode builds - as, in many cases, strings will be pure ASCII
even in international contexts (because many strings store non-human
language data, such as XML fragments, HTTP headers, JSON-encoded data,
etc.). We also hope that it will, for the same reasons, increase CPU
cache efficiency on non-trivial applications. The memory usage of
Python 3.3 is two to three times smaller than Python 3.2, and a little
bit better than Python 2.7, on a Django benchmark (see the PEP for
details)."

[1] http://docs.python.org/3/whatsnew/3.3.html
drago01
2012-11-15 13:19:43 UTC
Permalink
Post by drago01
Post by Alek Paunov
So, to me it seems natural joining all above together to start thinking
for replacing the classic python C bindings with thin textual or
bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
dependencies like Anaconda (thanks to lupa, Lua objects behaves like
Python ones in Python). The result will be order of magnitude easier for
maintenance (towards the APIs evolution) and faster code.
LuaJIT is currently not available for our secondary architectures, so this
would need fixing first. ("not available" meaning exactly that, there is no
fallback to a slow C interpreter mode or something like that.)
It's also not clear how significant the actual memory savings will be
because some of the Lua OO frameworks have quite a bit of overhead. And the
eventually arriving abrt plugin will bring some weight with it, too.
This is nothing against Lua or LuaJIT. I just don't think that switching
programming languages will magically reduce resource consumption. This has
to be an explicit design goal, or else it is not likely to happen. I'm sure
it's possible to write resource-conserving Python code, too. It's just the
baseline overhead from the run-time system which is unavoidable, but that's
fairly small for Python (apparently less than 2 MB of unshared RSS per
process).
Porting yum to python3 might be an easier task ... from the python 3.3
"The storage of Unicode strings now depends on the highest codepoint
pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per codepoint;
BMP strings (U+0000-U+FFFF) use 2 bytes per codepoint;
non-BMP strings (U+10000-U+10FFFF) use 4 bytes per codepoint.
The net effect is that for most applications, memory usage of string
storage should decrease significantly - especially compared to former
wide unicode builds - as, in many cases, strings will be pure ASCII
even in international contexts (because many strings store non-human
language data, such as XML fragments, HTTP headers, JSON-encoded data,
etc.). We also hope that it will, for the same reasons, increase CPU
cache efficiency on non-trivial applications. The memory usage of
Python 3.3 is two to three times smaller than Python 3.2, and a little
bit better than Python 2.7, on a Django benchmark (see the PEP for
details)."
[1] http://docs.python.org/3/whatsnew/3.3.html
Oh seems like the large savings are just compared to python 3.2 ... so
would not really help.
Alek Paunov
2012-11-15 18:44:01 UTC
Permalink
Hi Florian,
Post by Florian Weimer
Post by Alek Paunov
So, to me it seems natural joining all above together to start thinking
for replacing the classic python C bindings with thin textual or
bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
dependencies like Anaconda (thanks to lupa, Lua objects behaves like
Python ones in Python). The result will be order of magnitude easier for
maintenance (towards the APIs evolution) and faster code.
LuaJIT is currently not available for our secondary architectures, so
this would need fixing first. ("not available" meaning exactly that,
there is no fallback to a slow C interpreter mode or something like that.)
Oh, I completely forgot the secondary architectures (besides ARM).
Florian, you follow LuaJIT - what other architectures (except the
clearly missing s390) should be covered ?.
Post by Florian Weimer
It's also not clear how significant the actual memory savings will be
because some of the Lua OO frameworks have quite a bit of overhead. And
the eventually arriving abrt plugin will bring some weight with it, too.
This is nothing against Lua or LuaJIT. I just don't think that
switching programming languages will magically reduce resource
consumption. This has to be an explicit design goal, or else it is not
likely to happen. I'm sure it's possible to write resource-conserving
Python code, too. It's just the baseline overhead from the run-time
system which is unavoidable, but that's fairly small for Python
(apparently less than 2 MB of unshared RSS per process).
Apparently, my pseudo-English is not enough for a normal communication :-(.

I am commenting on the possibility for deploying LuaJIT FFI as a partial
replacement (problematic + future) of the hand written or SWIG Python
bindings in the first place, not switching the system language from
Python to Lua. In a system tool/daemon written in Python:

* Python loads libluajit.so (import lupa; from lupa import LuaRuntime;
...)

* The C libraries (the targets of the binding) are described as LuaJIT
FFI style bindings in simplified C [my primary point for that is
correctness (avoiding the memory leaks and reference counting problems)
and lowering the burden of the APIs evolution maintenance, not lowering
the general memory footprint]

* The current OO style Python wrappers (the higher level part of the
classical C bindings) are implemented with the Lua metamethods over
plain C objects (without any Lua OO frameworks)

* Python calls the LuaJIT C lib object wrappers (as close as possible
to the Python APIs exposed by the current bindings on top of the Python
C API).

Kind Regards,
Alek
Florian Weimer
2012-11-16 09:49:33 UTC
Permalink
Post by Alek Paunov
Post by Florian Weimer
LuaJIT is currently not available for our secondary architectures, so
this would need fixing first. ("not available" meaning exactly that,
there is no fallback to a slow C interpreter mode or something like that.)
Oh, I completely forgot the secondary architectures (besides ARM).
Florian, you follow LuaJIT - what other architectures (except the
clearly missing s390) should be covered ?.
ppc64 isn't supported, either.
Post by Alek Paunov
I am commenting on the possibility for deploying LuaJIT FFI as a partial
replacement (problematic + future) of the hand written or SWIG Python
bindings in the first place, not switching the system language from
* Python loads libluajit.so (import lupa; from lupa import LuaRuntime;
...)
And this will reduce memory overhead? I doubt it. Every C object will
be wrapped in a high-level Python object, a low-level Python object
(provided by lupa), a Lua registry slot (used by lupa), and an FFI
wrapper (provided by LuaJIT). Correctness requires that the GIL (or
some other lock) is held while C functions are called through the FFI
because LuaJIT is not reentrant.

There are some open issues, too. LuaJIT hasn't yet found a way to
handle C preprocessor defines (which are often used to provide
constants). There is not attempt at all to deal with conflicting
declarations. If two modules need definitions for time_t or struct
timespec, one of them will fail with an "attempt to redefine" error.

(Don't get me wrong, LuaJIT is a great piece of work.)
--
Florian Weimer / Red Hat Product Security Team
Przemek Klosowski
2012-11-15 19:26:39 UTC
Permalink
Post by Miloslav Trmač
Post by Kevin Kofler
Right. We need to stop writing core system components in scripting
languages!
Well, there _are_ significant advantages to using a higer-level
language than C.[1] Using one of the higher-level languages as a
primary development language on par with C often increases the quality
of the software and the time to develop it, in return for an
acceptable loss of speed.\
.......
Post by Miloslav Trmač
- Automatic memory management, with all associated lack of bugs and
simplified code
- Much easier access to higher-level (and more efficient!) data
structures. C programs frequently use linked lists instead of e.g.
hash tables simply because maintaining hash tables and the associated
memory allocation is just too complex.
+1!

I imagine that Kevin objects to the interpreter-induced slowness of
scripting, but quite often it's not even relevant because the code
doesn't run that much. To Miloslav's list of advantages, I would add
access to better libraries (e.g. network programming in Python or Tcl is
much easier than socket()-based code). There's the force multiplier
effect of higher level libraries allowing better algorithms, which often
trumps any advantages from compiled code.

If there's a _measurable_ speed issue, all modern interpreters have an
easy 'native method' interface, i.e. way of linking compiled code
functions that implement time-critical parts, if they are too slow for
interpreting. Let's avoid premature optimization--otherwise, taking this
idea to the extreme, one could argue for rewriting system services in
optimized assembly.

Remember also that data is code: any config files could be seen as tiny
specialized interpreted languages, so it's not like you can avoid
interpretation anyway. Perhaps embracing scripting in system components
would have a desirable side effect of more uniform config file syntax
because it's just easier to use some standard form like JSON or XML..
Kevin Kofler
2012-11-17 17:26:44 UTC
Permalink
Post by Przemek Klosowski
Remember also that data is code: any config files could be seen as tiny
specialized interpreted languages, so it's not like you can avoid
interpretation anyway.
That's a bad view of things, it leads to WTFs like PolicyKit using rules
written in JavaScript. A simple key-value store is not and should not be
Turing-complete, or even anywhere near Turing-complete. The logic needs to
be in the native code, not in the configuration.
Post by Przemek Klosowski
Perhaps embracing scripting in system components would have a desirable
side effect of more uniform config file syntax because it's just easier to
use some standard form like JSON or XML..
Just standardize on one of the many C-based INI-style file parsing
libraries.

Kevin Kofler
Przemek Klosowski
2012-11-20 17:52:30 UTC
Permalink
Post by Kevin Kofler
Post by Przemek Klosowski
Remember also that data is code: any config files could be seen as tiny
specialized interpreted languages, so it's not like you can avoid
interpretation anyway.
That's a bad view of things, it leads to WTFs like PolicyKit using rules
written in JavaScript. A simple key-value store is not and should not be
Turing-complete, or even anywhere near Turing-complete. The logic needs to
be in the native code, not in the configuration.
You say it yourself---if you need Turing, you have to put him in
somewhere. You argue that the complex logic HAS to go in the native
code, but that is inflexible. PolicyKit needed flexibility so they used
weird data; they combined inflexibility of native code with
inscrutability of complex logic in interpreted input. Plus they still
have a run-time and maintenance overhead of the interpreter, which they
were trying to avoid by designing a native code implementation.

Interpreters do not preclude simple data: they just scale better, from
simple linear declarative data to complex, Turing-cranking swamp. The
only argument against it is runtime overhead, which isn't a problem in
many, if not most, cases.
Rahul Sundaram
2012-11-20 18:39:59 UTC
Permalink
Hi

" Interpreters do not preclude simple data: they just scale better, from
simple linear declarative data to complex, Turing-cranking swamp. The only
argument against it is runtime overhead, which isn't a problem in many, if
not most, cases."

The problem is that, this design makes complex things possible but it
doesn't do enough to keep simple things easy.

Rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121120/48db33ef/attachment.html>
Przemek Klosowski
2012-11-20 20:39:27 UTC
Permalink
Post by Rahul Sundaram
" Interpreters do not preclude simple data: they just scale better, from
simple linear declarative data to complex, Turing-cranking swamp.
The problem is that, this design makes complex things possible but it
doesn't do enough to keep simple things easy.
It can be made simple, if you look at it the right way. One wouldn't
start with a generic interpreter, but rather evaluate the config script
in a domain-specific context. For instance we can define domain-specific
commands: the following could then be a valid script in Python, Perl,
shell, TCL, etc:

UUID "5a2541d6-f610-4455-9169-614971541b64"
NM_CONTROLLED "yes"
BOOTPROTO "dhcp"
DEVICE "em1"
ONBOOT yes
TYPE Ethernet
Rahul Sundaram
2012-11-21 10:50:44 UTC
Permalink
Hi
It can be made simple, if you look at it the right way. One wouldn't start
with a generic interpreter, but rather evaluate the config script in a
domain-specific context.
I think you just agreed in a roundabout way It *could* be made simple
perhaps but there is no documentation or best practices document that
covers how sysadmins are supposed to handle the simple configuration needs
like the pkla format did. Replacing pkla with the ability to run
Javascript doesn't make them feel powerful. It just makes them uneasy.
Sysadmins are not typically familiar with Javascript. A lot of GNOME
developers these days are but that doesn't translate into a good design for
PolicyKit.

Rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121121/a6bd20e3/attachment-0001.html>
Przemek Klosowski
2012-11-21 15:01:07 UTC
Permalink
Post by Przemek Klosowski
It can be made simple, if you look at it the right way. One wouldn't
start with a generic interpreter, but rather evaluate the config
script in a domain-specific context.
I think you just agreed in a roundabout way It *could* be made simple
perhaps but there is no documentation or best practices document that
covers how sysadmins are supposed to handle the simple configuration
needs like the pkla format did. Replacing pkla with the ability to run
Javascript doesn't make them feel powerful. It just makes them uneasy.
Sysadmins are not typically familiar with Javascript. A lot of GNOME
developers these days are but that doesn't translate into a good design
for PolicyKit.
I agree that this is not optimal--I think what happened was the original
design intended to embed the logic in the compiled code, then the
requirement for flexibility required the scripting capability, and the
Javascript library was available for embedding, and PK ended with an
interpreter bag sutured to its side.

This discussion started by someone's criticism of scripting, and I
started to point out that scripting is useful and often necessary, so it
finds its way in anyway---so if we just accepted this and embraced
scripting, and tried to settle on a common, best practices way, it is
possible to do it so that it's simple in the straightforward cases, and
scales up to complex logic if needed.
Kevin Kofler
2012-11-22 02:54:00 UTC
Permalink
Post by Przemek Klosowski
You say it yourself---if you need Turing, you have to put him in
somewhere. You argue that the complex logic HAS to go in the native
code, but that is inflexible. PolicyKit needed flexibility so they used
weird data; they combined inflexibility of native code with
inscrutability of complex logic in interpreted input. Plus they still
have a run-time and maintenance overhead of the interpreter, which they
were trying to avoid by designing a native code implementation.
Interpreters do not preclude simple data: they just scale better, from
simple linear declarative data to complex, Turing-cranking swamp. The
only argument against it is runtime overhead, which isn't a problem in
many, if not most, cases.
Runtime overhead is always a problem. It makes things pointlessly
inefficient. Plus, JavaScript configuration is a PITA to maintain
(especially for the user, who's normally NOT a programmer!). Configuration
has no business being scriptable. A key-value store is not and should not be
a script. Logic needs to be in code, not in configuration. Soft coding
sucks: http://thedailywtf.com/Articles/Soft_Coding.aspx

Kevin Kofler
Richard W.M. Jones
2012-11-22 20:07:06 UTC
Permalink
Post by Przemek Klosowski
Interpreters do not preclude simple data: they just scale better,
from simple linear declarative data to complex, Turing-cranking
swamp. The only argument against it is runtime overhead, which isn't
a problem in many, if not most, cases.
It's NOT the only argument against it. Having Turing-complete
configuration files makes it impossible to have other programs parse
and understand the configuration. Programs including:

- OpenSCAP, or any other security scanner
- libvirt (hello, old Xen's python config files)
- multiple libguestfs tools like virt-sysprep
- Augeas and all the tools that use it

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
Simon Lukasik
2012-11-23 10:43:01 UTC
Permalink
Post by Richard W.M. Jones
Post by Przemek Klosowski
Interpreters do not preclude simple data: they just scale better,
from simple linear declarative data to complex, Turing-cranking
swamp. The only argument against it is runtime overhead, which isn't
a problem in many, if not most, cases.
It's NOT the only argument against it. Having Turing-complete
configuration files makes it impossible to have other programs parse
- OpenSCAP, or any other security scanner
- libvirt (hello, old Xen's python config files)
- multiple libguestfs tools like virt-sysprep
- Augeas and all the tools that use it
Moreover, If the application (polkit) uses its embedded interpreter to
assess configuration and the scanner (OpenSCAP) uses it's own way how to
assess it (even if it differs in a version of the interpreter). --> It
only opens door for very subtle bugs.

Which leads me to thinking that the applications (which use Turing
complete languages for configuration) shall provide a comprehensive API
to query the configuration.
Post by Richard W.M. Jones
Rich.
--
Simon Lukasik
Richard W.M. Jones
2012-11-25 16:12:36 UTC
Permalink
Post by Simon Lukasik
Post by Richard W.M. Jones
Post by Przemek Klosowski
Interpreters do not preclude simple data: they just scale better,
from simple linear declarative data to complex, Turing-cranking
swamp. The only argument against it is runtime overhead, which isn't
a problem in many, if not most, cases.
It's NOT the only argument against it. Having Turing-complete
configuration files makes it impossible to have other programs parse
- OpenSCAP, or any other security scanner
- libvirt (hello, old Xen's python config files)
- multiple libguestfs tools like virt-sysprep
- Augeas and all the tools that use it
Moreover, If the application (polkit) uses its embedded interpreter to
assess configuration and the scanner (OpenSCAP) uses it's own way how to
assess it (even if it differs in a version of the interpreter). --> It
only opens door for very subtle bugs.
Which leads me to thinking that the applications (which use Turing
complete languages for configuration) shall provide a comprehensive API
to query the configuration.
This isn't going to work for SCAP. SCAP (or more specifically, OVAL)
is a standardized XML schema for assessing the configuration of
systems. Steve will correct me if I'm wrong here, but I don't believe
there's no room for it to be calling out to arbitrary custom
libraries.
http://oval.mitre.org/language/index.html
http://oval.mitre.org/language/about/definition.html

Like it or not, this sort of scanning is extremely useful for cloud
administrators who want to be able to automatically scan disk images
uploaded from non-trusted sources and find out whether they contain
vulnerabilities. The requirements for configuration files to be
simple and non-Turing-complete are not going to go away.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
Richard W.M. Jones
2012-11-25 16:17:33 UTC
Permalink
Post by Richard W.M. Jones
systems. Steve will correct me if I'm wrong here, but I don't believe
there's no room for it to be calling out to arbitrary custom
s/no/any/
Post by Richard W.M. Jones
libraries.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
Simon Lukasik
2012-11-26 11:09:31 UTC
Permalink
Post by Richard W.M. Jones
Post by Simon Lukasik
Post by Richard W.M. Jones
Post by Przemek Klosowski
Interpreters do not preclude simple data: they just scale better,
from simple linear declarative data to complex, Turing-cranking
swamp. The only argument against it is runtime overhead, which isn't
a problem in many, if not most, cases.
It's NOT the only argument against it. Having Turing-complete
configuration files makes it impossible to have other programs parse
- OpenSCAP, or any other security scanner
- libvirt (hello, old Xen's python config files)
- multiple libguestfs tools like virt-sysprep
- Augeas and all the tools that use it
Moreover, If the application (polkit) uses its embedded interpreter to
assess configuration and the scanner (OpenSCAP) uses it's own way how to
assess it (even if it differs in a version of the interpreter). --> It
only opens door for very subtle bugs.
Which leads me to thinking that the applications (which use Turing
complete languages for configuration) shall provide a comprehensive API
to query the configuration.
This isn't going to work for SCAP. SCAP (or more specifically, OVAL)
is a standardized XML schema for assessing the configuration of
systems. Steve will correct me if I'm wrong here, but I don't believe
there's no room for it to be calling out to arbitrary custom
libraries.
Sure, there is no room for *arbitrary* and custom libraries. On the
other hand, there are libraries which are already in use. Consider
librpm which is even namely regarded by OVAL specification. Thus, If
there was comprehensive interface to query polkit configuration, I could
imagine SCAP using it in similar way how it uses librpm.

But my point was more like: If there was an comprehensive interface to
read and write that configuration, there couldn't be a complain about
compliance checks.
Post by Richard W.M. Jones
http://oval.mitre.org/language/index.html
http://oval.mitre.org/language/about/definition.html
Like it or not, this sort of scanning is extremely useful for cloud
administrators who want to be able to automatically scan disk images
uploaded from non-trusted sources and find out whether they contain
vulnerabilities. The requirements for configuration files to be
simple and non-Turing-complete are not going to go away.
Rich.
--
Simon Lukasik
Richard W.M. Jones
2012-11-26 11:15:01 UTC
Permalink
You're asking for something very open-ended, and which is not even
possible to implement in general -- what if the inspecting operating
system is Linux, and the inspected operating system needs custom Mac OS X
libraries?

In any case, the correct place to request this is with the SCAP
standards team.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
Simon Lukasik
2012-11-28 12:43:48 UTC
Permalink
Post by Richard W.M. Jones
You're asking for something very open-ended, and which is not even
possible to implement in general -- what if the inspecting operating
system is Linux, and the inspected operating system needs custom Mac OS X
libraries?
Perhaps I am just dense, but I can't follow you.

I can't see how this is related to my latest post. And I can't see what
leads you to think that cross-platform scanning is feasible today
(standard-wise and performance-wise).
Post by Richard W.M. Jones
In any case, the correct place to request this is with the SCAP
standards team.
Which request?

Where did I requested something which has been bind to the SCAP? I only
asked for an interface in a distribution.

Regards,
--
Simon Lukasik
Richard W.M. Jones
2012-11-28 13:50:44 UTC
Permalink
Post by Simon Lukasik
Post by Richard W.M. Jones
You're asking for something very open-ended, and which is not even
possible to implement in general -- what if the inspecting operating
system is Linux, and the inspected operating system needs custom Mac OS X
libraries?
Perhaps I am just dense, but I can't follow you.
I can't see how this is related to my latest post. And I can't see what
leads you to think that cross-platform scanning is feasible today
(standard-wise and performance-wise).
It's exactly what we plan to do with libguestfs & openscap.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
Alek Paunov
2012-11-28 19:17:20 UTC
Permalink
Hi Simon,
Post by Simon Lukasik
I can't see how this is related to my latest post. And I can't see what
leads you to think that cross-platform scanning is feasible today
(standard-wise and performance-wise).
Please check this very informative post from Steve (OVAL board member):

http://www.mail-archive.com/devel at lists.fedoraproject.org/msg50625.html

We (the camp arguing for "flexible" configurations) should identify
clever proposal at least conforming the following:

* Will be able to be "cold" (statically) assessed through current or
future OVAL (first and most important use-case - VM images evaluation,
second - augeas and alike on the live systems, third - custom auditing
agents)

* Will looks cool for the sysadmins (the config languages users) (I am
in doubt that any traditional scripting language will bring high score
on this criteria)

* Will enable desired (for the non-trivial cases) flexibility

Kind regards,
Alek

Ales Kozumplik
2012-11-12 11:06:26 UTC
Permalink
Post by Seth Vidal
Post by Kevin Kofler
Post by Richard W.M. Jones
- depends on Python stack
+1, we really need to get Python out of the minimal installation.
The focus should be on replacing the existing Python-based packages in the
minimum set (e.g. yum) by native replacements (e.g. zif). Adding more Python
stuff with additional Python dependencies is a step backwards.
I really don't understand why a core system component such as
firewalld is
implemented in Python!
Yum will likely be replaced with dnf afaik. I don't think zif is under
consideration at all.
-sv
DNF is going to be a replacement for Yum, but the underlying libraries
are C and I can imagine a simple, fairly straightforward tool on top of
them that does the job for minimal install, all in C.

Ales
Kevin Kofler
2012-11-12 18:52:53 UTC
Permalink
Post by Seth Vidal
Yum will likely be replaced with dnf afaik. I don't think zif is under
consideration at all.
That's exactly what I'm complaining about. Dnf is no improvement over yum at
all, zif would bring real advantages through the simple fact that it's
native code, not Python. Native C code is faster, requires less memory and
is less likely to break when the system is broken (and thus more likely to
be usable to repair it). Zif also interoperates much better with PackageKit
(on which our package management UIs, gnome-packagekit and Apper, are
based), being implemented in the same language by the same primary developer
and designed with PackageKit in mind (whereas yum happily breaks APIs used
by PackageKit, see e.g. the repo.str() case; apparently, you don't even test
your changes with PackageKit!).

Kevin Kofler
Richard Vickery
2012-11-12 21:41:02 UTC
Permalink
If dnf is no improvement, then I'd rather we stick with yum; messing with
something new just means spending time that I don't have trying to learn
that new command. This is incredibly cumbersome. If at all possible, please
stay with yum.
Post by Kevin Kofler
Post by Seth Vidal
Yum will likely be replaced with dnf afaik. I don't think zif is under
consideration at all.
That's exactly what I'm complaining about. Dnf is no improvement over yum at
all, zif would bring real advantages through the simple fact that it's
native code, not Python. Native C code is faster, requires less memory and
is less likely to break when the system is broken (and thus more likely to
be usable to repair it). Zif also interoperates much better with PackageKit
(on which our package management UIs, gnome-packagekit and Apper, are
based), being implemented in the same language by the same primary developer
and designed with PackageKit in mind (whereas yum happily breaks APIs used
by PackageKit, see e.g. the repo.str() case; apparently, you don't even test
your changes with PackageKit!).
Kevin Kofler
--
devel mailing list
devel at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121112/deccdd52/attachment.html>
Chris Adams
2012-11-12 21:46:11 UTC
Permalink
Post by Richard Vickery
If dnf is no improvement, then I'd rather we stick with yum; messing with
something new just means spending time that I don't have trying to learn
that new command. This is incredibly cumbersome. If at all possible, please
stay with yum.
You should learn about something before you criticize it:

https://fedoraproject.org/wiki/Features/DNF

dnf is currently implementing a (subset of) existing yum commands.
--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
Richard Vickery
2012-11-12 23:14:30 UTC
Permalink
Change still frightens people to varying degree s., and many busy end-users
may not have time to read pages in order to learn a new command
Post by Richard Vickery
Post by Richard Vickery
If dnf is no improvement, then I'd rather we stick with yum; messing with
something new just means spending time that I don't have trying to learn
that new command. This is incredibly cumbersome. If at all possible,
please
Post by Richard Vickery
stay with yum.
https://fedoraproject.org/wiki/Features/DNF
dnf is currently implementing a (subset of) existing yum commands.
--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
--
devel mailing list
devel at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121112/ba9fc62a/attachment.html>
Chris Adams
2012-11-12 23:59:00 UTC
Permalink
Post by Richard Vickery
Change still frightens people to varying degree s., and many busy end-users
may not have time to read pages in order to learn a new command
It is in _development_ and is just in F18 as a preview. If/when it is
ready to replace yum, it may get a symlink from /usr/bin/yum.
--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
Jan Zelený
2012-11-13 07:48:32 UTC
Permalink
Post by Chris Adams
Post by Richard Vickery
Change still frightens people to varying degree s., and many busy end-users
may not have time to read pages in order to learn a new command
It is in _development_ and is just in F18 as a preview. If/when it is
ready to replace yum, it may get a symlink from /usr/bin/yum.
That's the plan.

Thanks
Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121113/a51898cc/attachment.sig>
Rahul Sundaram
2012-11-15 20:29:19 UTC
Permalink
Post by Jan Zelený
Post by Chris Adams
Post by Richard Vickery
Change still frightens people to varying degree s., and many busy end-users
may not have time to read pages in order to learn a new command
It is in _development_ and is just in F18 as a preview. If/when it is
ready to replace yum, it may get a symlink from /usr/bin/yum.
That's the plan.
Can you guys post all of the plan in this list? (as a sep announcement) I
am not sure the broader Fedora community has any real idea what dnf or the
seemingly newly created packaging team plans are

Rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121115/2e18d6f3/attachment.html>
Matthew Miller
2012-11-12 19:53:01 UTC
Permalink
Post by Kevin Kofler
I really don't understand why a core system component such as firewalld is
implemented in Python!
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.


And for reducing space use: I think it might also be nice to break python
2to3 and idle out of the python-libs package.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Toshio Kuratomi
2012-11-12 20:42:20 UTC
Permalink
Post by Matthew Miller
Post by Kevin Kofler
I really don't understand why a core system component such as firewalld is
implemented in Python!
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
And for reducing space use: I think it might also be nice to break python
2to3 and idle out of the python-libs package.
Eh....

2to3 is used in code besides the 2to3 command line tool.

idlelib I aven't seen being used but...

These are libraries provided by the main python distribution whether or not
they're frequently used. If this is something we want to pursuse, perhaps
we really want to look at all of the python stdlib and decide if there's
other things that are not popular and we'd wish to make into subpackages.

This is not a one-time piece of work, however -- there'd be ongoing
maintainance -- for instance, if we broke the 2to3 module into its own
subpackage and then a different python-stdlib module started importing it
we'd have to spot that new inter-dependency and move 2to3 back into the
stdlib.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20121112/65ce63cc/attachment.sig>
Thomas Woerner
2012-11-13 11:53:10 UTC
Permalink
Post by Matthew Miller
Post by Kevin Kofler
I really don't understand why a core system component such as firewalld is
implemented in Python!
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated. Also each
call would take a lot longer, which is bad for example for libvirt.
Post by Matthew Miller
And for reducing space use: I think it might also be nice to break python
2to3 and idle out of the python-libs package.
Richard W.M. Jones
2012-11-13 13:19:36 UTC
Permalink
Post by Thomas Woerner
Post by Matthew Miller
Post by Kevin Kofler
I really don't understand why a core system component such as firewalld is
implemented in Python!
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated.
Surely it could persist it somewhere?

However I doubt that changing firewalld to be dbus activated would
make any actual difference to memory usage. We do have a thing called
swap ... The reason I object to firewalld being written in Python are
the excessive dependencies this adds for a component that should be
included in a minimal/core set of packages.

OCaml, you see ...

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
Tomasz Torcz
2012-11-13 13:28:17 UTC
Permalink
Post by Richard W.M. Jones
Post by Thomas Woerner
Post by Matthew Miller
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated.
Surely it could persist it somewhere?
Like in the actual netfilter rules?
--
Tomasz Torcz "God, root, what's the difference?"
xmpp: zdzichubg at chrome.pl "God is more forgiving."
Matthew Miller
2012-11-13 14:46:31 UTC
Permalink
Post by Tomasz Torcz
Post by Richard W.M. Jones
Post by Thomas Woerner
Post by Matthew Miller
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated.
Surely it could persist it somewhere?
Like in the actual netfilter rules?
Yes.

It has to be able to save internal state *somehow*, because if restarting
the service breaks everything, we're not gaining much over the old way, are
we? Plus, for a critical service like this, the service needs to be designed
to be as robust as possible in situations where it might crash or get killed
arbitrarily.

And for things like the ten-second-temporary rule, it could hang around for
a while.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Thomas Woerner
2012-11-13 16:28:42 UTC
Permalink
Post by Matthew Miller
Post by Tomasz Torcz
Post by Richard W.M. Jones
Post by Thomas Woerner
Post by Matthew Miller
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated.
Surely it could persist it somewhere?
Like in the actual netfilter rules?
Yes.
It has to be able to save internal state *somehow*, because if restarting
the service breaks everything, we're not gaining much over the old way, are
we? Plus, for a critical service like this, the service needs to be designed
to be as robust as possible in situations where it might crash or get killed
arbitrarily.
With the old static firewall model every firewall change was a complete
firewall recreate with conntrack loss. With firewalld changes to the
firewall are done dynamically and conntrack is preserved.

If you want to recreate rules, use reload. If you restart the service
with systemd, the servce gets stopped and started again, so you will
loose internal state. This is how services are working.
Post by Matthew Miller
And for things like the ten-second-temporary rule, it could hang around for
a while.
It is using glib timeouts for this, it is not hanging around and blocking.
Matthew Miller
2012-11-13 16:36:38 UTC
Permalink
Post by Thomas Woerner
If you want to recreate rules, use reload. If you restart the
service with systemd, the servce gets stopped and started again, so
you will loose internal state. This is how services are working.
I understand that some services work that way. However, I don't think that
this is the best design for a firewall service. Is there some way to force
the internal state to be recorded?

Let's say there is a security fix for the firewall service which needs to be
applied. The daemon will need to be reloaded. Is this now not possible?
Post by Thomas Woerner
Post by Matthew Miller
And for things like the ten-second-temporary rule, it could hang around for
a while.
It is using glib timeouts for this, it is not hanging around and blocking.
Sorry, this comment lost context: I didn't mean that the timeout
implementation was poor. I meant that if the service were dbus activated, it
could stay running if it continued to have things to do, and exit (maybe
after a brief wait) if not.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Thomas Woerner
2012-11-13 17:03:18 UTC
Permalink
Post by Matthew Miller
Post by Thomas Woerner
If you want to recreate rules, use reload. If you restart the
service with systemd, the servce gets stopped and started again, so
you will loose internal state. This is how services are working.
I understand that some services work that way. However, I don't think that
this is the best design for a firewall service. Is there some way to force
the internal state to be recorded?
Let's say there is a security fix for the firewall service which needs to be
applied. The daemon will need to be reloaded. Is this now not possible?
Some services work that way? Only some? If you have a security fix, you
have to restart every service to get the new code.

Firewalld is not able to save the state for a later start.
Post by Matthew Miller
Post by Thomas Woerner
Post by Matthew Miller
And for things like the ten-second-temporary rule, it could hang around for
a while.
It is using glib timeouts for this, it is not hanging around and blocking.
Sorry, this comment lost context: I didn't mean that the timeout
implementation was poor. I meant that if the service were dbus activated, it
could stay running if it continued to have things to do, and exit (maybe
after a brief wait) if not.
The security team asked me not to make firewalld a D-BUS driven
mechanism, because of security concerns and also because of SELinux.

Additionally every load of the mechanism could have to load modified or
removed configuration files. So how should it get to the same state then
again? How should it react on and reflect configuration changes? So it
would have to write out everything - the state and all configuration
files. I am sorry, but this is overkill and a most likely a source of
big problems.
Matthew Miller
2012-11-13 17:21:44 UTC
Permalink
Post by Thomas Woerner
Post by Matthew Miller
I understand that some services work that way. However, I don't think that
this is the best design for a firewall service. Is there some way to force
the internal state to be recorded?
Let's say there is a security fix for the firewall service which needs to be
applied. The daemon will need to be reloaded. Is this now not possible?
Some services work that way? Only some? If you have a security fix,
you have to restart every service to get the new code.
Correct. Some services save their state immediately to disk when there is a
critical change, and others do so on shutdown.
Post by Thomas Woerner
Firewalld is not able to save the state for a later start.
Okay. I think it needs to be able to, before it's ready for prime-time.
Post by Thomas Woerner
Post by Matthew Miller
Sorry, this comment lost context: I didn't mean that the timeout
implementation was poor. I meant that if the service were dbus activated,
it could stay running if it continued to have things to do, and exit
(maybe after a brief wait) if not.
The security team asked me not to make firewalld a D-BUS driven
mechanism, because of security concerns and also because of SELinux.
SELinux and D-Bus can work together.

Can you elaborate on the security concerns?
Post by Thomas Woerner
Additionally every load of the mechanism could have to load modified
or removed configuration files. So how should it get to the same
state then again? How should it react on and reflect configuration
changes? So it would have to write out everything - the state and
all configuration files. I am sorry, but this is overkill and a most
likely a source of big problems.
Any firewall service clearly needs to do these things whether or not it is
D-Bus activated.

Given that, doing it all the time is not significantly more work, and
actually likely to be more robust, because the code will be the normal path
and exercised all the time, not a special case.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Lennart Poettering
2012-11-15 00:53:32 UTC
Permalink
Post by Thomas Woerner
The security team asked me not to make firewalld a D-BUS driven
mechanism, because of security concerns and also because of SELinux.
Uh? If you write a new D-Bus service and want to use bus activation,
then you should SystemdService= in the dbus service snippet, so that the
actual activation is done by systemd. If you do that, then the effective
difference regarding the execution environment of the service is next to
zero, you just have two different triggers ("start this on boot"
vs. "start this when dbus tells systemd to").

In fact, all system bus services should be configured to defer
activation to systemd, so that all services regardless how they are
triggered are executed in the same clean execution environment, and can
be manipulated with the same commands (systemctl
stop/kill/restart/mask/...) as any other services.

In fact, even if you write a bus service that is started at boot, you
should *always* make it bus activatable as well. Why? For two reasons:
you make things robust that way, as the service is restarted when needed
should it have died, and you enable parallelized boot-up, since services
requiring your services don't have to be ordered explicitly after yours,
but can just talk to your service name and dbus will queue all messages
until your service is actually up.

Hence: please make your service bus-activatable. In all cases. There are
no security concerns, and you make things faster and more robust by
doing this.
Post by Thomas Woerner
Additionally every load of the mechanism could have to load modified
or removed configuration files. So how should it get to the same
state then again? How should it react on and reflect configuration
changes? So it would have to write out everything - the state and
all configuration files. I am sorry, but this is overkill and a most
likely a source of big problems.
Actually, no, it's not overkill, it's good coding practice. Most of
systemd's various little services (not all though) are written that they
can be killed and restarted at any time without losing any state (such
as logind, udevd, timedated, localed, hostnamed, ...). In fact even
systemd itself can exit and be reexecuted while recovering almost all
previous state. (In fact, currently the one exception of the systemd
services that can't just be killed without negative effect is journald,
but we are working on that).

It's really not that hard. I think it's a really good rule to make all
low-level system daemons work that way, to keep things robust and
resource usage minimal.

Lennart
--
Lennart Poettering - Red Hat, Inc.
Matthew Miller
2012-11-15 15:06:16 UTC
Permalink
Post by Lennart Poettering
In fact, all system bus services should be configured to defer
activation to systemd, so that all services regardless how they are
triggered are executed in the same clean execution environment, and can
be manipulated with the same commands (systemctl
stop/kill/restart/mask/...) as any other services.
[...]
Post by Lennart Poettering
It's really not that hard. I think it's a really good rule to make all
low-level system daemons work that way, to keep things robust and
resource usage minimal.
I was looking for resources on systemd and dbus activation. I realized that
the documentation here is very scarce -- there's this tutorial on just dbus
activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>,
but it seems like it might make a nice "systemd for Developers", if you
feel inspired at any point.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Matej Cepl
2012-11-15 15:15:25 UTC
Permalink
Post by Matthew Miller
I was looking for resources on systemd and dbus activation. I realized that
the documentation here is very scarce -- there's this tutorial on just dbus
activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>,
but it seems like it might make a nice "systemd for Developers", if you
feel inspired at any point.
There is a lot of stuff on Lennart’s blog http://0pointer.de/blog/

Matěj
Matthew Miller
2012-11-15 16:16:45 UTC
Permalink
Post by Matej Cepl
Post by Matthew Miller
I was looking for resources on systemd and dbus activation. I realized that
the documentation here is very scarce -- there's this tutorial on just dbus
activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>,
but it seems like it might make a nice "systemd for Developers", if you
feel inspired at any point.
There is a lot of stuff on Lennart’s blog http://0pointer.de/blog/
There *is* a lot of stuff on Lennart's blog. But unless I missed something,
none of the articles are about D-Bus activation.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Lennart Poettering
2012-11-15 18:20:56 UTC
Permalink
Post by Matthew Miller
Post by Lennart Poettering
In fact, all system bus services should be configured to defer
activation to systemd, so that all services regardless how they are
triggered are executed in the same clean execution environment, and can
be manipulated with the same commands (systemctl
stop/kill/restart/mask/...) as any other services.
[...]
Post by Lennart Poettering
It's really not that hard. I think it's a really good rule to make all
low-level system daemons work that way, to keep things robust and
resource usage minimal.
I was looking for resources on systemd and dbus activation. I realized that
the documentation here is very scarce -- there's this tutorial on just dbus
activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>,
but it seems like it might make a nice "systemd for Developers", if you
feel inspired at any point.
There's indeed not much documentation around. This is primarily because
we were kinda waiting for kernel dbus to materialize which would
probably shuffle a few things around in this area.

Here's the gist of it:

1) In /usr/share/dbus-1/system-services/*.service use the
SystemdService= setting to declare that activation of a specific
D-Bus service should be done with systemd.

2) Basically, that's already it. However, here's a twist: instead of
directly specifiying the systemd service file name I recommend
specifying the name of a symlink which point to the actual
service, and is available only if the service is actually
enabled. Then, add Alias= for this symlink name to [Install] in the
unit file. This way you can now enable/disable the service and this
affects bus activation as administrators would expect.

Here's an example for Avahi:

In /usr/share/dbus-1/system-services/org.freedesktop.Avahi.service:

[D-BUS Service]
Name=org.freedesktop.Avahi
SystemdService=dbus-org.freedesktop.Avahi.service

And in /usr/lib/systemd/system/avahi-daemon.service:

[Unit]
Description=Avahi mDNS/DNS-SD Stack
Requires=avahi-daemon.socket

[Service]
Type=dbus
BusName=org.freedesktop.Avahi
ExecStart=/usr/sbin/avahi-daemon -s
ExecReload=/usr/sbin/avahi-daemon -r
NotifyAccess=main

[Install]
WantedBy=multi-user.target
Also=avahi-daemon.socket
Alias=dbus-org.freedesktop.Avahi.service

The important line here is the last, which has the effect that the
symlink /etc/systemd/system/dbus-org.freedesktop.Avahi.service is
managed by "systemctl enable" and "systemctl disable" so that the bus
activation only works when avahi is actually enabled.

Naming the symlinks "dbus-xyz.service" (with xyz being the actual bus
name) is a just a recommendation, nothing is requiring that...

Lennart
--
Lennart Poettering - Red Hat, Inc.
Dennis Jacobfeuerborn
2012-11-13 17:16:49 UTC
Permalink
Post by Thomas Woerner
Post by Matthew Miller
Post by Tomasz Torcz
Post by Richard W.M. Jones
Post by Thomas Woerner
Post by Matthew Miller
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated.
Surely it could persist it somewhere?
Like in the actual netfilter rules?
Yes.
It has to be able to save internal state *somehow*, because if restarting
the service breaks everything, we're not gaining much over the old way, are
we? Plus, for a critical service like this, the service needs to be designed
to be as robust as possible in situations where it might crash or get killed
arbitrarily.
With the old static firewall model every firewall change was a complete
firewall recreate with conntrack loss. With firewalld changes to the
firewall are done dynamically and conntrack is preserved.
That's not correct. You can modify the firewall just fine without
restarting it.

Regards,
Dennis
Thomas Woerner
2012-11-13 17:37:37 UTC
Permalink
Post by Dennis Jacobfeuerborn
Post by Thomas Woerner
Post by Matthew Miller
Post by Tomasz Torcz
Post by Richard W.M. Jones
Post by Thomas Woerner
Post by Matthew Miller
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated.
Surely it could persist it somewhere?
Like in the actual netfilter rules?
Yes.
It has to be able to save internal state *somehow*, because if restarting
the service breaks everything, we're not gaining much over the old way, are
we? Plus, for a critical service like this, the service needs to be designed
to be as robust as possible in situations where it might crash or get killed
arbitrarily.
With the old static firewall model every firewall change was a complete
firewall recreate with conntrack loss. With firewalld changes to the
firewall are done dynamically and conntrack is preserved.
That's not correct. You can modify the firewall just fine without
restarting it.
This is related to system-config-firewall/lokkit. You are right, if you
are using iptables directly then you do not have this limitation.
firewalld is a replacement for s-c-fw/lokkit.
Post by Dennis Jacobfeuerborn
Regards,
Dennis
Matthew Miller
2012-11-13 17:43:13 UTC
Permalink
Post by Thomas Woerner
Post by Dennis Jacobfeuerborn
That's not correct. You can modify the firewall just fine without
restarting it.
This is related to system-config-firewall/lokkit. You are right, if
you are using iptables directly then you do not have this
limitation. firewalld is a replacement for s-c-fw/lokkit.
This is not what it says in the feature page at
https://fedoraproject.org/wiki/Features/firewalld-default#Detailed_Description

That says:

The services iptables, iptables-ipv6 and ebtables will be replaced by
firewalld. system-config-firewall in it's [sic] current form will also be
replaced.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Bill Nottingham
2012-11-13 20:40:54 UTC
Permalink
Post by Matthew Miller
Post by Thomas Woerner
Post by Dennis Jacobfeuerborn
That's not correct. You can modify the firewall just fine without
restarting it.
This is related to system-config-firewall/lokkit. You are right, if
you are using iptables directly then you do not have this
limitation. firewalld is a replacement for s-c-fw/lokkit.
This is not what it says in the feature page at
https://fedoraproject.org/wiki/Features/firewalld-default#Detailed_Description
The services iptables, iptables-ipv6 and ebtables will be replaced by
firewalld. system-config-firewall in it's [sic] current form will also be
replaced.
Replaced in the default configuration - you obviously shouldn't be running
firewalld and the static firewall scripts at the same time, so enabling them
in combination would be a bad idea.

Bill
Tomasz Torcz
2012-11-13 21:00:09 UTC
Permalink
Post by Dennis Jacobfeuerborn
Post by Thomas Woerner
Post by Matthew Miller
Post by Tomasz Torcz
Post by Richard W.M. Jones
Post by Thomas Woerner
Post by Matthew Miller
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It would loose internal state if it would be D-BUS activated.
Surely it could persist it somewhere?
Like in the actual netfilter rules?
Yes.
It has to be able to save internal state *somehow*, because if restarting
the service breaks everything, we're not gaining much over the old way, are
we? Plus, for a critical service like this, the service needs to be designed
to be as robust as possible in situations where it might crash or get killed
arbitrarily.
With the old static firewall model every firewall change was a complete
firewall recreate with conntrack loss. With firewalld changes to the
firewall are done dynamically and conntrack is preserved.
That's not correct. You can modify the firewall just fine without
restarting it.
We are drifting away from the question. We are trying to understand what
is the blocker for firewalld going away when not needed. There is some “state”
to be preserved. Apart from timers for “temporary” rules, what kind of
state firewalld has? State that cannot be expressed by netfiler rules in
kernel, of course.
--
Tomasz Torcz To co nierealne -- tutaj jest normalne.
xmpp: zdzichubg at chrome.pl Ziomale na ÅŒycie mają tu patenty specjalne.
Pádraig Brady
2012-11-13 14:41:44 UTC
Permalink
Post by Matthew Miller
Post by Kevin Kofler
I really don't understand why a core system component such as firewalld is
implemented in Python!
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
It could be argued that python is more suited to long lived programs:

$ time /bin/true
real 0m0.002s
$ time python -c True
real 0m0.049s
$ time python3 -c True
real 0m0.165s
Post by Matthew Miller
And for reducing space use: I think it might also be nice to break python
2to3 and idle out of the python-libs package.
splitting python-libs (25MB here), seems worthwhile.
python-libs can bb changed to a subpackage that just depends on
various split subpackages, and then as needed various packages
like yum etc. can adjust dependencies to require just the
subpackages they need.

I remember doing a dist of python for an embedded system
that was 2.1MB in total and was enough to support cherrypy.

cheers,
Pádraig.
Matthew Miller
2012-11-13 14:50:52 UTC
Permalink
Post by Pádraig Brady
Post by Matthew Miller
And for reducing space use: I think it might also be nice to break python
2to3 and idle out of the python-libs package.
splitting python-libs (25MB here), seems worthwhile.
python-libs can bb changed to a subpackage that just depends on
various split subpackages, and then as needed various packages
like yum etc. can adjust dependencies to require just the
subpackages they need.
Yep. I pick those two because they're both big and both the kind of lib
where if your package needs them, you probably know it.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Chris Adams
2012-11-13 14:51:59 UTC
Permalink
Post by Pádraig Brady
$ time /bin/true
real 0m0.002s
$ time python -c True
real 0m0.049s
Aside from that being a meaningless, worst case example, an "overhead"
of .047 seconds is hardly worth worrying about unless the command in
question is run multiple times per minute, all the time.

And besides:

$ time perl -e 1
real 0m0.008s

Perl is 6 times faster than Python! :)
--
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
Matthew Miller
2012-11-13 15:00:42 UTC
Permalink
Post by Chris Adams
Post by Pádraig Brady
$ time /bin/true
real 0m0.002s
$ time python -c True
real 0m0.049s
Aside from that being a meaningless, worst case example, an "overhead"
of .047 seconds is hardly worth worrying about unless the command in
question is run multiple times per minute, all the time.
And in that case I assume it would stay active.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm at fedoraproject.org>
Richard W.M. Jones
2012-11-13 16:35:59 UTC
Permalink
Post by Pádraig Brady
Post by Matthew Miller
Post by Kevin Kofler
I really don't understand why a core system component such as firewalld is
implemented in Python!
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.
$ time /bin/true
real 0m0.002s
Hmmm:

$ echo '' > true.ml
$ ocamlopt.opt true.ml -o true
$ time ./true

real 0m0.002s
user 0m0.000s
sys 0m0.001s

$ time /bin/true

real 0m0.001s
user 0m0.000s
sys 0m0.001s

This seems about right to me: Both ocamlopt & gcc generate native
x86-64 programs, but there's a small amount of overhead in the OCaml
binary (initializing the minor heap of the GC).

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
Roberto Ragusa
2012-11-14 08:47:16 UTC
Permalink
Post by Richard W.M. Jones
Post by Pádraig Brady
$ time /bin/true
real 0m0.002s
$ echo '' > true.ml
$ ocamlopt.opt true.ml -o true
$ time ./true
real 0m0.002s
user 0m0.000s
sys 0m0.001s
$ time /bin/true
real 0m0.001s
user 0m0.000s
sys 0m0.001s
This seems about right to me: Both ocamlopt & gcc generate native
x86-64 programs, but there's a small amount of overhead in the OCaml
binary (initializing the minor heap of the GC).
Just for fun, let's see who is the worst.

$ cat >JavaTrue.java
public class JavaTrue {
public static void main(String[] args) {
System.exit(0);
}
}
$ javac JavaTrue.java
$ time java -cp . JavaTrue

real 0m0.071s
user 0m0.037s
sys 0m0.035s

Hmmm. Slow and with a lot of sys...
Let's try compiling it to native.

$ gcj -o JavaTrue JavaTrue.java --main=JavaTrue
$ time ./JavaTrue

real 0m0.026s
user 0m0.016s
sys 0m0.010s

Good improvement, but still quite slow.

(OT: what a pity gcj is abandoned... great project)
--
Roberto Ragusa mail at robertoragusa.it
Björn Persson
2012-11-14 18:38:09 UTC
Permalink
Post by Roberto Ragusa
Just for fun, let's see who is the worst.
Challenge accepted. :-)

$ cat nothing.adb
procedure Nothing is
begin
null;
end Nothing;
$ gnatmake nothing.adb -o true
gcc -c nothing.adb
gnatbind -x nothing.ali
gnatlink nothing.ali -o true
$ time ./true

real 0m0.001s
user 0m0.000s
sys 0m0.000s

Björn Persson
Alek Paunov
2012-11-14 23:25:32 UTC
Permalink
Post by Richard W.M. Jones
$ echo '' > true.ml
$ ocamlopt.opt true.ml -o true
$ time ./true
real 0m0.002s
user 0m0.000s
sys 0m0.001s
time luajit -e "require'os'; os.exit(42)"

real 0m0.001s
user 0m0.000s
sys 0m0.000s

But, check here for a far more important feature:

http://luajit.org/ext_ffi_tutorial.html#zlib
Post by Richard W.M. Jones
This seems about right to me: Both ocamlopt & gcc generate native
x86-64 programs, but there's a small amount of overhead in the OCaml
binary (initializing the minor heap of the GC).
luajit too (it is one of the fastest compilers)

I will try to formulate an approach for Fedora to benefit from this
little miracle in additional note (SWIG bindings problems above in the
thread).

Kind Regards,
Alek
Kevin Kofler
2012-11-15 02:32:02 UTC
Permalink
Post by Alek Paunov
Post by Richard W.M. Jones
This seems about right to me: Both ocamlopt & gcc generate native
x86-64 programs, but there's a small amount of overhead in the OCaml
binary (initializing the minor heap of the GC).
luajit too (it is one of the fastest compilers)
Unlike the others, it generates the native code at runtime (Just In Time),
so there is a performance penalty (especially for nontrivial programs) for
the (JIT) compilation which gcc and ocamlopt won't have. The quality of the
generated code could also be a problem (for nonempty programs): Optimized
compilation takes time! So there's a tradeoff between the time of the JIT
compilation and the time of the actual execution, a very fast JIT is not
necessarily optimal because it may be missing important optimizations of the
compiled code.

In other words: Try nontrivial benchmarks before claiming victory.

Kevin Kofler
Alek Paunov
2012-11-15 03:04:02 UTC
Permalink
Post by Kevin Kofler
Unlike the others, it generates the native code at runtime (Just In Time),
so there is a performance penalty (especially for nontrivial programs) for
the (JIT) compilation which gcc and ocamlopt won't have. The quality of the
generated code could also be a problem (for nonempty programs): Optimized
compilation takes time! So there's a tradeoff between the time of the JIT
compilation and the time of the actual execution, a very fast JIT is not
necessarily optimal because it may be missing important optimizations of the
compiled code.
All above is correct, with the small nuance that all practically
significant slowdowns are always result of repeatable code paths (loops,
recursions, etc) but these lands to near optimal native code after the
first few iterations with this concrete implementation (Mike Pall is
well known with his in-depth theoretical background).
Post by Kevin Kofler
In other words: Try nontrivial benchmarks before claiming victory.
I am. Check GSL shell for example. Nobody claims any victories
(especially against gcc). The point is, that this concrete JIT is
probably the fastest existing one and brings bunch of advantages at the
cost of negligible footprint (compared with v8 for example).

Have you played with LuaJIT apps already ... or speaking from common
sense POV ?
Continue reading on narkive:
Loading...