glibc in Fedora rawhide now split by langpacks.

[Carlos O'Donell]

The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).

Thanks to Mike Fabian for the great work!

Bug 1238406 - Glibc locale subpackaging
https://bugzilla.redhat.com/show_bug.cgi?id=1238406

Changes/Glibc locale subpackaging
https://fedoraproject.org//wiki/Changes/Glibc_locale_subpackaging

Notes:
- If you don't use langpacks you won't get any locales
except C, POSIX and C.UTF-8.
- If you don't want to use langpacks just install the
glibc langpack you want manually.
e.g. dnf install glibc-langpack-en
- Anaconda should take care of everything for you via
langpacks.
- We missed the fact that for the F23->F24 transition
`dnf system-upgrade` should install glibc-all-langpacks
(meta package) to give F23 users a smooth transition
with all the locales they had before, but once in F24
the langpacks will be used.
Filed:
Bug 1312103 - “dnf system-upgrade” needs to install
“glibc-all-langpacks” when upgrading from f23 to f24
https://bugzilla.redhat.com/show_bug.cgi?id=1312103

Cheers,
Carlos.

[Vít Ondruch]

You also forget to mention what Rawhide users are supposed to do.

I assume that I should do:

# dnf install langpacks-cs

to have only czech language available and English should be available by
default, right?


Vít

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).
Thanks to Mike Fabian for the great work!
Bug 1238406 - Glibc locale subpackaging
https://bugzilla.redhat.com/show_bug.cgi?id=1238406
Changes/Glibc locale subpackaging
https://fedoraproject.org//wiki/Changes/Glibc_locale_subpackaging
- If you don't use langpacks you won't get any locales
except C, POSIX and C.UTF-8.
- If you don't want to use langpacks just install the
glibc langpack you want manually.
e.g. dnf install glibc-langpack-en
- Anaconda should take care of everything for you via
langpacks.
- We missed the fact that for the F23->F24 transition
`dnf system-upgrade` should install glibc-all-langpacks
(meta package) to give F23 users a smooth transition
with all the locales they had before, but once in F24
the langpacks will be used.
Bug 1312103 - “dnf system-upgrade” needs to install
“glibc-all-langpacks” when upgrading from f23 to f24
https://bugzilla.redhat.com/show_bug.cgi?id=1312103
Cheers,
Carlos.
--
devel mailing list

[Carlos O'Donell]

Post by Vít Ondruch
You also forget to mention what Rawhide users are supposed to do.
# dnf install langpacks-cs
to have only czech language available and English should be available by
default, right?

No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove. So for F24
you get nothing by default, and dnf langpacks plugin does everything
for you in the background.

If you're on rawhide, and you don't use langpacks, you can install
the appropriate langpack by hand with `dnf install glibc-langpack-cs`.

Cheers,
Carlos

[Miroslav Suchý]

Post by Carlos O'Donell
No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove.

For this reasons we have weak dependencies. I guess that:
Recommends: glibc-all-langpacks
can do the work (install or by default, can be safely removed). Or
Suggests: glibc-all-langpacks
so it is not installed by default, but users can get the information that it can enhance the usability of glibc.

[Stephen Gallagher]

Post by Miroslav Suchý

Post by Carlos O'Donell
No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove.

Recommends: glibc-all-langpacks
can do the work (install or by default, can be safely removed). Or
Suggests: glibc-all-langpacks
so it is not installed by default, but users can get the information that it can enhance the usability of glibc.

Yeah, I think the best approach would be to have all the langpacks offer a
virtual Provides: glibc-langpack and have the main package Requires:
glibc-langpack and Suggests: glibc-all-langpacks. The net result would be that
unless a specific langpack was chosen, you'd end up with all of them to satisfy
the requirement. (This would also unbreak the upgrades without needing a patch
to dnf system-upgrade)

[Zbigniew Jędrzejewski-Szmek]

Post by Stephen Gallagher

Post by Miroslav Suchý

Post by Carlos O'Donell
No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove.

Recommends: glibc-all-langpacks
can do the work (install or by default, can be safely removed). Or
Suggests: glibc-all-langpacks
so it is not installed by default, but users can get the information that it can enhance the usability of glibc.

Yeah, I think the best approach would be to have all the langpacks offer a
glibc-langpack and Suggests: glibc-all-langpacks. The net result would be that
unless a specific langpack was chosen, you'd end up with all of them to satisfy
the requirement. (This would also unbreak the upgrades without needing a patch
to dnf system-upgrade)

That would be a much better solution. dnf-system-upgrade so far didn't
have any special handling for specific packages (except for a check
that the kernel is in the upgrade transaction) and there no mechanism
like this is implemented.

Zbyszek

[Carlos O'Donell]

Post by Zbigniew Jędrzejewski-Szmek

Post by Stephen Gallagher

Post by Miroslav Suchý

Post by Carlos O'Donell
No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove.

Recommends: glibc-all-langpacks
can do the work (install or by default, can be safely removed). Or
Suggests: glibc-all-langpacks
so it is not installed by default, but users can get the information that it can enhance the usability of glibc.

Yeah, I think the best approach would be to have all the langpacks offer a
glibc-langpack and Suggests: glibc-all-langpacks. The net result would be that
unless a specific langpack was chosen, you'd end up with all of them to satisfy
the requirement. (This would also unbreak the upgrades without needing a patch
to dnf system-upgrade)

That would be a much better solution. dnf-system-upgrade so far didn't
have any special handling for specific packages (except for a check
that the kernel is in the upgrade transaction) and there no mechanism
like this is implemented.

We are investigating switching to this. It shouldn't be a problem, and
with a quick rebuild of glibc in F24 I think we'll be ready. I just need
to run through the testing quickly on Monday.

All of our langpacks are auto-generated based on glibc supported locales
so there is no grunt work in changing, just testing again that it works
smoothly.

Cheers,
Carlos.

[Adam Williamson]

Post by Carlos O'Donell

Post by Zbigniew Jędrzejewski-Szmek

Post by Stephen Gallagher

Post by Carlos O'Donell
No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove.

  Recommends: glibc-all-langpacks
can do the work (install or by default, can be safely removed). Or
  Suggests: glibc-all-langpacks
so it is not installed by default, but users can get the
information that it can enhance the usability of glibc.

Yeah, I think the best approach would be to have all the langpacks offer a
glibc-langpack and Suggests: glibc-all-langpacks. The net result would be that
unless a specific langpack was chosen, you'd end up with all of them to satisfy
the requirement. (This would also unbreak the upgrades without needing a patch
to dnf system-upgrade)

That would be a much better solution. dnf-system-upgrade so far didn't
have any special handling for specific packages (except for a check
that the kernel is in the upgrade transaction) and there no mechanism
like this is implemented.

We are investigating switching to this. It shouldn't be a problem, and
with a quick rebuild of glibc in F24 I think we'll be ready. I just need
to run through the testing quickly on Monday.
All of our langpacks are auto-generated based on glibc supported locales
so there is no grunt work in changing, just testing again that it works
smoothly.

I have filed https://bugzilla.redhat.com/show_bug.cgi?id=1312956 and
nominated it as a Beta blocker, for the issue of all locales
disappearing on upgrade.

[Lars Seipel]

Post by Stephen Gallagher
Yeah, I think the best approach would be to have all the langpacks offer a
glibc-langpack and Suggests: glibc-all-langpacks.

This would force the installation of at least one langpack, no? The C,
POSIX and *C.UTF-8* locales are builtin, so for many systems it is very
reasonable to run without any language pack installed.

[Stephen Gallagher]

Post by Lars Seipel

Post by Stephen Gallagher
Yeah, I think the best approach would be to have all the langpacks offer a
glibc-langpack and Suggests: glibc-all-langpacks.

This would force the installation of at least one langpack, no? The C,
POSIX and *C.UTF-8* locales are builtin, so for many systems it is very
reasonable to run without any language pack installed.

Yeah, the workaround we're doing there is to have a glibc-minimal-langpacks subpackage that satisfies the requirement but contains no actual files. It's a bit of a hack, but not a particularly awful one.

[Dominik 'Rathann' Mierzejewski]

Post by Stephen Gallagher

Post by Lars Seipel

Post by Stephen Gallagher
Yeah, I think the best approach would be to have all the langpacks offer a
glibc-langpack and Suggests: glibc-all-langpacks.

This would force the installation of at least one langpack, no? The C,
POSIX and *C.UTF-8* locales are builtin, so for many systems it is very
reasonable to run without any language pack installed.

Yeah, the workaround we're doing there is to have a glibc-minimal-langpacks
subpackage that satisfies the requirement but contains no actual files.
It's a bit of a hack, but not a particularly awful one.

Why a subpackage? Shouldn't the main glibc package (or glibc-common)
simply provide the C, C.UTF-8 and POSIX langpacks?

Regards,
Dominik

[Stephen Gallagher]

Post by Dominik 'Rathann' Mierzejewski

Post by Stephen Gallagher

Post by Lars Seipel

Post by Stephen Gallagher
Yeah, I think the best approach would be to have all the langpacks offer a
glibc-langpack and Suggests: glibc-all-langpacks.

This would force the installation of at least one langpack, no? The C,
POSIX and *C.UTF-8* locales are builtin, so for many systems it is very
reasonable to run without any language pack installed.

Yeah, the workaround we're doing there is to have a glibc-minimal-langpacks
subpackage that satisfies the requirement but contains no actual files.
It's a bit of a hack, but not a particularly awful one.

Why a subpackage? Shouldn't the main glibc package (or glibc-common)
simply provide the C, C.UTF-8 and POSIX langpacks?

The glibc-common package *does* provide the C, C.UTF-8 and POSIX langpacks. The
reason for the subpackage is just a hack to fix some issues with upgrades.

We needed a way to ensure that upgrades would pull in the complete set of
langpacks, since we had no way of knowing which ones were actually being used.
The way we did this was to add a "Requires: glibc-langpack" to the glibc package
and a "Suggests: glibc-all-langpacks" as well. Then we added "Provides:
glibc-langpack" to all of the subpackages that provide a language. However, this
resulted in not being able to restrict it to JUST the built-ins, so we added a
special, empty glibc-minimal-langpacks subpackage whose purpose is just to
satisfy the "Requires: glib-langpack" dependency. It contains no content and
doesn't add anything to the hard drive except the RPM database.

[Vít Ondruch]

Post by Carlos O'Donell

Post by Vít Ondruch
You also forget to mention what Rawhide users are supposed to do.
# dnf install langpacks-cs
to have only czech language available and English should be available by
default, right?

No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove. So for F24
you get nothing by default, and dnf langpacks plugin does everything
for you in the background.
If you're on rawhide, and you don't use langpacks, you can install
the appropriate langpack by hand with `dnf install glibc-langpack-cs`.

This is very badly communicated I must say. I really don't understand,
why the suggested steps are not:

1) dnf install langpack-[your-language]
2) dnf update glibc\*

This should be the workflow and not "you use" or "you don't use". This
is not something which will be glibc specific. The
langpack-[your-language] is precondition to make this work, not to
suggest installing glibc-langapack-[your-language] or even
glibc-all-langpacks.

I am bit annoyed by this, because doing just "dnf update" will break the
system in a way that gnome-terminal won't start anymore, DNF is totally
busted etc. so simple instructions would be really appreciated.


Vít

[Carlos O'Donell]

Post by Vít Ondruch

Post by Carlos O'Donell

Post by Vít Ondruch
You also forget to mention what Rawhide users are supposed to do.
# dnf install langpacks-cs
to have only czech language available and English should be available by
default, right?

No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove. So for F24
you get nothing by default, and dnf langpacks plugin does everything
for you in the background.
If you're on rawhide, and you don't use langpacks, you can install
the appropriate langpack by hand with `dnf install glibc-langpack-cs`.

This is very badly communicated I must say. I really don't understand,
1) dnf install langpack-[your-language]
2) dnf update glibc\*
This should be the workflow and not "you use" or "you don't use". This
is not something which will be glibc specific. The
langpack-[your-language] is precondition to make this work, not to
suggest installing glibc-langapack-[your-language] or even
glibc-all-langpacks.
I am bit annoyed by this, because doing just "dnf update" will break the
system in a way that gnome-terminal won't start anymore, DNF is totally
busted etc. so simple instructions would be really appreciated.

I agree, and I apologize. We're working on an rpm dependency based resolution
process which will fix this shortly.

Cheers,
Carlos.

[Vít Ondruch]

Post by Carlos O'Donell

Post by Vít Ondruch

Post by Carlos O'Donell

Post by Vít Ondruch
You also forget to mention what Rawhide users are supposed to do.
# dnf install langpacks-cs
to have only czech language available and English should be available by
default, right?

No languages are available by default, we did this because otherwise
you keep carrying forward locales that you can't remove. So for F24
you get nothing by default, and dnf langpacks plugin does everything
for you in the background.
If you're on rawhide, and you don't use langpacks, you can install
the appropriate langpack by hand with `dnf install glibc-langpack-cs`.

This is very badly communicated I must say. I really don't understand,
1) dnf install langpack-[your-language]
2) dnf update glibc\*
This should be the workflow and not "you use" or "you don't use". This
is not something which will be glibc specific. The
langpack-[your-language] is precondition to make this work, not to
suggest installing glibc-langapack-[your-language] or even
glibc-all-langpacks.
I am bit annoyed by this, because doing just "dnf update" will break the
system in a way that gnome-terminal won't start anymore, DNF is totally
busted etc. so simple instructions would be really appreciated.

I agree, and I apologize. We're working on an rpm dependency based resolution
process which will fix this shortly.

Although I was grumpy, I would also like to thank you for working on
this, since this is definitely step in the right direction. I appreciate
your effort. Thanks.

Vít

[Carlos O'Donell]

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).

Unfortunately we missed a piece of the puzzle here.

Bug 1303034 - rpm macro expansion works incorrectly
when looping over a long list using lua
https://bugzilla.redhat.com/show_bug.cgi?id=1303034

We need this fixed in F24 and pushed into the buildroots
to unblock building glibc.

Cheers,
Carlos.

[Parag Nemade]

Hi,

Post by Carlos O'Donell

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).

Unfortunately we missed a piece of the puzzle here.
Bug 1303034 - rpm macro expansion works incorrectly
when looping over a long list using lua
https://bugzilla.redhat.com/show_bug.cgi?id=1303034
We need this fixed in F24 and pushed into the buildroots
to unblock building glibc.

I thought this is already fixed in F24. At least I remember I tested
langpacks glibc.spec from mfabian's some old private git repository on
F24 machine. The rpm build I tested was rpm-4.13.0-0.rc1.23.fc24. But
now I see that f24 glibc is failing while comparing the SUPPORTED file
in %prep.


Regards,
Parag.

[Carlos O'Donell]

Post by Parag Nemade
Hi,

Post by Carlos O'Donell

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).

Unfortunately we missed a piece of the puzzle here.
Bug 1303034 - rpm macro expansion works incorrectly
when looping over a long list using lua
https://bugzilla.redhat.com/show_bug.cgi?id=1303034
We need this fixed in F24 and pushed into the buildroots
to unblock building glibc.

I thought this is already fixed in F24. At least I remember I tested
langpacks glibc.spec from mfabian's some old private git repository on
F24 machine. The rpm build I tested was rpm-4.13.0-0.rc1.23.fc24. But
now I see that f24 glibc is failing while comparing the SUPPORTED file
in %prep.

We didn't get that far in last nights build. I'll go look immediately.

Cheers,
Carlos.

[Richard Hughes]

Post by Carlos O'Donell
We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf.

What happens if the user isn't using DNF? For the workstation we have
to support users using just the graphical tools, and we can't rely on
command line tools for this kind of thing. I'm happy to add support
for installing lang packs into gnome-software but need to know how the
langpack plugin works so we can add proper support into the AppStream
metadata file. The idea being if the user changes the per-session
language we install the required langpacks automatically.

Can someone explain how the dnf langpack plugin gets the data on what
application subpackages to install? It would be easy if we could have
a virtual package that we could install for each locale as then we can
just put the metainfo file there for easy support in gnome-software.

Thanks,

Richard

[Peter Robinson]

Post by Richard Hughes
What happens if the user isn't using DNF? For the workstation we have
to support users using just the graphical tools, and we can't rely on
command line tools for this kind of thing. I'm happy to add support
for installing lang packs into gnome-software but need to know how the
langpack plugin works so we can add proper support into the AppStream
metadata file. The idea being if the user changes the per-session
language we install the required langpacks automatically.
Can someone explain how the dnf langpack plugin gets the data on what
application subpackages to install? It would be easy if we could have
a virtual package that we could install for each locale as then we can
just put the metainfo file there for easy support in gnome-software.

My understanding this is implementing using the same langpack
functionality that has been there for ages even in yum for all sorts
of lang support, such as dictionaries, grammar etc, it's just been
that glibc has never been able to split out the language support in
the past.

Peter

[Parag Nemade]

Hi Richard,

Post by Richard Hughes

Post by Carlos O'Donell
We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf.

What happens if the user isn't using DNF? For the workstation we have
to support users using just the graphical tools, and we can't rely on
command line tools for this kind of thing. I'm happy to add support
for installing lang packs into gnome-software but need to know how the
langpack plugin works so we can add proper support into the AppStream
metadata file. The idea being if the user changes the per-session
language we install the required langpacks automatically.
Can someone explain how the dnf langpack plugin gets the data on what
application subpackages to install? It would be easy if we could have
a virtual package that we could install for each locale as then we can
just put the metainfo file there for easy support in gnome-software.

Please have a look at New Langpacks Change[1] information. With this
change we are trying to remove the need of having dnf langpacks
plugin. The basic required things are already in F24+ but I have not
yet got time to add weak dependencies in all the required langpacks.
But I have modified basic required langpacks like hunspell, hyphen,
mythes. Then libreoffice, kde-l10n, gnome-getting-started-docs and
tesseract has been also built for this new langpacks change. I do have
created test page[2] for this new change.

I suppose all the langpacks changes will be complete by next week.
Please note that I specifically requested Workstation WG to add
dnf-langpacks package in comps but later on we decided to use weak
dependencies for langpacks. Now users need to just install
meta-packages like langpacks-<langcode> to get all the required
langpacks installed on the system. If needed we can remove
dnf-langpacks from Fedora Workstation installation but I will say
better wait till Fedora 24 Alpha release and see if new langpacks
implementation has worked fine by that time.

Regards,
Parag

[1] https://fedoraproject.org/wiki/Changes/LangpacksInstallationWithRPMWeakDependencies
[2] https://fedoraproject.org/wiki/QA:Langpacks

[Richard Hughes]

Post by Parag Nemade
Please have a look at New Langpacks Change[1] information.

Thanks for the information -- this looks a lot better than the current
logic and so a huge thanks for driving this. One more question: Would
you be against me adding (translated) metainfo files to each of the
langpack subpackages? This would allow me to add support in GNOME
Software very easily without any hacky code. I'm happy doing the work
if this sounds acceptable to you.

Richard

[Parag Nemade]

Hi,

Post by Richard Hughes

Post by Parag Nemade
Please have a look at New Langpacks Change[1] information.

Thanks for the information -- this looks a lot better than the current
logic and so a huge thanks for driving this. One more question: Would
you be against me adding (translated) metainfo files to each of the
langpack subpackages? This would allow me to add support in GNOME
Software very easily without any hacky code. I'm happy doing the work
if this sounds acceptable to you.

I am fine with you adding metainfo files to langpacks.spec file.

Regards,
Parag.

[Michael Catanzaro]

Post by Richard Hughes
What happens if the user isn't using DNF? For the workstation we have
to support users using just the graphical tools, and we can't rely on
command line tools for this kind of thing. I'm happy to add support
for installing lang packs into gnome-software but need to know how the
langpack plugin works so we can add proper support into the AppStream
metadata file. The idea being if the user changes the per-session
language we install the required langpacks automatically.
Can someone explain how the dnf langpack plugin gets the data on what
application subpackages to install? It would be easy if we could have
a virtual package that we could install for each locale as then we can
just put the metainfo file there for easy support in gnome-software.

We also need locale selection in gnome-control-center and gnome-
initial-setup to work. Any chance you'll work on that too, Richard?

Otherwise, we should be able to resolve this by adding glibc-all-
langpacks to the Workstation group in comps (correct Carlos?). It
defeats the point of this change, but releasing with broken locale
selection is not gonna happen.

Michael

[Carlos O'Donell]

Post by Michael Catanzaro

Post by Richard Hughes
What happens if the user isn't using DNF? For the workstation we have
to support users using just the graphical tools, and we can't rely on
command line tools for this kind of thing. I'm happy to add support
for installing lang packs into gnome-software but need to know how the
langpack plugin works so we can add proper support into the AppStream
metadata file. The idea being if the user changes the per-session
language we install the required langpacks automatically.
Can someone explain how the dnf langpack plugin gets the data on what
application subpackages to install? It would be easy if we could have
a virtual package that we could install for each locale as then we can
just put the metainfo file there for easy support in gnome-software.

We also need locale selection in gnome-control-center and gnome-
initial-setup to work. Any chance you'll work on that too, Richard?
Otherwise, we should be able to resolve this by adding glibc-all-
langpacks to the Workstation group in comps (correct Carlos?). It
defeats the point of this change, but releasing with broken locale
selection is not gonna happen.

Correct, 'glibc-all-langpacks' will solve the problem immediately.

We're working right now on the solution Stephen suggested which is
a weak ref/suggests which looks like it would let rpm handle the
upgrade properly.

Cheers,
Carlos.

[Michael Catanzaro]

Post by Carlos O'Donell
Correct, 'glibc-all-langpacks' will solve the problem immediately.

OK, I've added this to comps, to take care of new installs. If anyone
implements support for language pack installation in both gnome-
initial-setup and gnome-control-center, we can remove this.

Post by Carlos O'Donell
We're working right now on the solution Stephen suggested which is
a weak ref/suggests which looks like it would let rpm handle the
upgrade properly.

And I trust you guys will take care of upgrades. Thanks!

Michael

[Carlos O'Donell]

Post by Michael Catanzaro

Post by Carlos O'Donell
Correct, 'glibc-all-langpacks' will solve the problem immediately.

OK, I've added this to comps, to take care of new installs. If anyone
implements support for language pack installation in both gnome-
initial-setup and gnome-control-center, we can remove this.

Post by Carlos O'Donell
We're working right now on the solution Stephen suggested which is
a weak ref/suggests which looks like it would let rpm handle the
upgrade properly.

And I trust you guys will take care of upgrades. Thanks!

We've also had some questions about the performance and disk space
of the split up langpacks. So I'm going to work on a blog post to
help clarify this for everyone.

Cheers,
Carlos.

[Ruben Kerkhof]

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).
Thanks to Mike Fabian for the great work!

I'm now seeing this in mock / koji builds:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

For example https://kojipkgs.fedoraproject.org//packages/collectd/5.5.1/4.fc25/data/logs/armv7hl/build.log

It looks harmless, but could this be related to this change and is
this already a known issue?

Kind regards,

Ruben Kerkhof

[Petr Pisar]

Post by Ruben Kerkhof
perl: warning: Setting locale failed.
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Mock (or rpmbuild?) sets LANG to en_US.UTF-8 instead of C in order to
have UTF-8 locale when building packages because it was decided that
UTF-8 is the default (build) environment for Fedora.

With removing en_US.UTF-8 from the glibc and thus minimal build root,
all koji builds are misconfigured now because setlocale(3) on "en_US.UTF-8"
fails without the locale definition in the file system.

I assume changing the default build-time locale to C.UTF-8 could help.

-- Petr

[Carlos O'Donell]

Post by Petr Pisar

Post by Ruben Kerkhof
perl: warning: Setting locale failed.
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Mock (or rpmbuild?) sets LANG to en_US.UTF-8 instead of C in order to
have UTF-8 locale when building packages because it was decided that
UTF-8 is the default (build) environment for Fedora.
With removing en_US.UTF-8 from the glibc and thus minimal build root,
all koji builds are misconfigured now because setlocale(3) on "en_US.UTF-8"
fails without the locale definition in the file system.
I assume changing the default build-time locale to C.UTF-8 could help.

The glibc team guarantees you will have C.UTF-8 always.

It should be the default.

This is indeed related to the langpack split up and we are going
to fix this shortly.

Cheers,
Carlos.

[Dan Callaghan]

Post by Carlos O'Donell

Post by Petr Pisar
Mock (or rpmbuild?) sets LANG to en_US.UTF-8 instead of C in order to
have UTF-8 locale when building packages because it was decided that
UTF-8 is the default (build) environment for Fedora.
With removing en_US.UTF-8 from the glibc and thus minimal build root,
all koji builds are misconfigured now because setlocale(3) on "en_US.UTF-8"
fails without the locale definition in the file system.

One of my Python packages' tests failed in Koschei due to (I think) the
same issue. The tests were failing in Python 3 because now sys.stdout is
encoded as ASCII whereas it (presumably) should be UTF-8.

Post by Carlos O'Donell

Post by Petr Pisar
I assume changing the default build-time locale to C.UTF-8 could help.

The glibc team guarantees you will have C.UTF-8 always.
It should be the default.
This is indeed related to the langpack split up and we are going
to fix this shortly.

When you say it will be fixed, do you mean fixed by changing Koji to run
all builds in C.UTF-8 instead of en_US.UTF-8? Or fixed by ensuring that
en_US.UTF-8 is available in the build environment (by adding it to the
Koji build group I guess)? Or some other fix?

[Adam Williamson]

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).
Thanks to Mike Fabian for the great work!

This did completely break anaconda (and thus also broke live image and
Cloud image creation).

https://bugzilla.redhat.com/show_bug.cgi?id=1312607

[Igor Gnatenko]

And also I was not able to start gnome-terminal and some other apps. I had
to install glibc-langpack-en.

Post by Adam Williamson

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).
Thanks to Mike Fabian for the great work!

This did completely break anaconda (and thus also broke live image and
Cloud image creation).
https://bugzilla.redhat.com/show_bug.cgi?id=1312607
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
--
devel mailing list

[Adam Williamson]

Post by Igor Gnatenko
And also I was not able to start gnome-terminal and some other apps. I had
to install glibc-langpack-en.

Different case. The issue of how to handle upgrade of existing installs
is already being discussed upthread. I was pointing out a different
problem.

[Richard W.M. Jones]

Post by Igor Gnatenko
And also I was not able to start gnome-terminal and some other apps. I had
to install glibc-langpack-en.

libvirtd also fails to start up:

https://bugzilla.redhat.com/show_bug.cgi?id=1312688

I'm investigating it now.

Rich.

[Richard W.M. Jones]

Post by Richard W.M. Jones

Post by Igor Gnatenko
And also I was not able to start gnome-terminal and some other apps. I had
to install glibc-langpack-en.

https://bugzilla.redhat.com/show_bug.cgi?id=1312688
I'm investigating it now.

Is it OK to ignore the return value of setlocale()?

Rich.

[Carlos O'Donell]

Post by Richard W.M. Jones

Post by Richard W.M. Jones

Post by Igor Gnatenko
And also I was not able to start gnome-terminal and some other apps. I had
to install glibc-langpack-en.

https://bugzilla.redhat.com/show_bug.cgi?id=1312688
I'm investigating it now.

Is it OK to ignore the return value of setlocale()?

Yes and no.

It's a quality issue.

Without the locale your localizations like dates may be in
formats the user doesn't expect.

It's up to you to decide what quality you consider "critical."

We are going to fix this shortly in F24+ though so that more
locales are installed during the upgrade.

Cheers,
Carlos.

[Carlos O'Donell]

Post by Adam Williamson

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).
Thanks to Mike Fabian for the great work!

This did completely break anaconda (and thus also broke live image and
Cloud image creation).
https://bugzilla.redhat.com/show_bug.cgi?id=1312607

The glibc team is working to put out an ASAP update that
would use a slightly tweaked set of rpm dependencies to
ensure glibc-all-lanpacks is pulled in by default in more cases.

We appreciate your patience.

Cheers,
Carlos.

[Adam Williamson]

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).

Also in unpredictable fallout: on upgraded systems, ssh stops working
until you install a langpack. At least it did for me.

[Carlos O'Donell]

Post by Adam Williamson

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).

Also in unpredictable fallout: on upgraded systems, ssh stops working
until you install a langpack. At least it did for me.

That should never happen. What failed?

Cheers,
Carlos.

[Adam Williamson]

Post by Carlos O'Donell

Post by Adam Williamson

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf. This
greatly reduces the size of a glibc install from 130MB
down to a couple of megs. It drops support for the hacky
%_inst_langs feature, and relies entirely on langpack
support (it was one or the other).

Also in unpredictable fallout: on upgraded systems, ssh stops working
until you install a langpack. At least it did for me.

That should never happen. What failed?

IIRC it would simply terminate claiming the remote host had dropped the
connection. There wasn't any useful information even with -vvvv. I just
kinda guessed it must have something to do with the langpacks,
installed glibc-langpack-en, and it started working again immediately.
I can't reproduce the problem with 'LC_ALL=C ssh (somewhere)', so I
can't immediately dig into it any more.

[Florian Weimer]

Post by Adam Williamson
IIRC it would simply terminate claiming the remote host had dropped the
connection. There wasn't any useful information even with -vvvv. I just
kinda guessed it must have something to do with the langpacks,
installed glibc-langpack-en, and it started working again immediately.
I can't reproduce the problem with 'LC_ALL=C ssh (somewhere)', so I
can't immediately dig into it any more.

Could you try with “LC_ALL=de_DE.UTF-8 ssh …”? Assuming that
de_DE.UTF-8 is not installed on the target.

Other distributions already lack a full complement of locales in default
installations, and most software works reasonably well in the presence
of unsupported locale settings. It's certainly something we need to
iron out.

Florian

[Carlos O'Donell]

Post by Florian Weimer

Post by Adam Williamson
IIRC it would simply terminate claiming the remote host had dropped the
connection. There wasn't any useful information even with -vvvv. I just
kinda guessed it must have something to do with the langpacks,
installed glibc-langpack-en, and it started working again immediately.
I can't reproduce the problem with 'LC_ALL=C ssh (somewhere)', so I
can't immediately dig into it any more.

Could you try with “LC_ALL=de_DE.UTF-8 ssh …”? Assuming that
de_DE.UTF-8 is not installed on the target.
Other distributions already lack a full complement of locales in default
installations, and most software works reasonably well in the presence
of unsupported locale settings. It's certainly something we need to
iron out.

Agreed. The similar gnome-terminal failure should be fixed e.g.
https://bugzilla.redhat.com/show_bug.cgi?id=1312690

Cheers,
Carlos.

[Adam Williamson]

Post by Adam Williamson
IIRC it would simply terminate claiming the remote host had dropped the
connection. There wasn't any useful information even with -vvvv. I just
kinda guessed it must have something to do with the langpacks,
installed glibc-langpack-en, and it started working again immediately.
I can't reproduce the problem with 'LC_ALL=C ssh (somewhere)', so I
can't immediately dig into it any more.

Could you try with “LC_ALL=de_DE.UTF-8 ssh …”?  Assuming that
de_DE.UTF-8 is not installed on the target.
Other distributions already lack a full complement of locales in default
installations, and most software works reasonably well in the presence
of unsupported locale settings.  It's certainly something we need to
iron out.

Yeah, that does the trick:

[***@adam comps (master %)]$ LC_ALL=de_DE.UTF-8 ssh www
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
ssh_exchange_identification: Connection closed by remote host
[***@adam comps (master %)]$ ssh www
Last login: Mon Feb 29 08:16:59 2016 from 192.168.1.5

note this is on Rawhide, not F24. Dunno if it affects F24.

[Adam Williamson]

Post by Adam Williamson

Post by Adam Williamson
IIRC it would simply terminate claiming the remote host had dropped the
connection. There wasn't any useful information even with -vvvv. I just
kinda guessed it must have something to do with the langpacks,
installed glibc-langpack-en, and it started working again immediately.
I can't reproduce the problem with 'LC_ALL=C ssh (somewhere)', so I
can't immediately dig into it any more.

Could you try with “LC_ALL=de_DE.UTF-8 ssh …”?  Assuming that
de_DE.UTF-8 is not installed on the target.
Other distributions already lack a full complement of locales in default
installations, and most software works reasonably well in the presence
of unsupported locale settings.  It's certainly something we need to
iron out.

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
ssh_exchange_identification: Connection closed by remote host
Last login: Mon Feb 29 08:16:59 2016 from 192.168.1.5
note this is on Rawhide, not F24. Dunno if it affects F24.

https://bugzilla.redhat.com/show_bug.cgi?id=1313041

[Bruno Wolff III]

Upgrading to this version of glibc also breaks postgres if you have a
database that uses something other than the C locale. Installing appropriate
language packs and restarting fix the issue.

[Carlos O'Donell]

Post by Carlos O'Donell
Bug 1238406 - Glibc locale subpackaging
https://bugzilla.redhat.com/show_bug.cgi?id=1238406
Changes/Glibc locale subpackaging
https://fedoraproject.org//wiki/Changes/Glibc_locale_subpackaging

With glibc-2.23.90-3.fc25 and glibc-2.23.1-5.fc24 we have now fixed
the issue in several important ways.

The `glibc-all-langpacks` continues to be the correct way to get all
locales, but now things are a bit smoother.

(1) Virtual provides to make upgrade smoother.

All glibc languages packs now provide `glibc-langpack`, that includes
the glibc-all-langpacks (all language support).

The core runtime depends on `glibc-langpack` and so when upgrading
on systems without dnf, yum, or langpack plugin support, plain rpm
dependency solving via suggests will install `glibc-all-langpack`
for you and provide all the existing locales you previously had.

This makes for a complete rpm solution with no other support
required.

This means that upgrading from a non-langpack supporting F23
to any future langpack supporting distribution will upgrade
you to the newest set of supported locales.

(2) Optimized "all languages" performance.

In addition the `glibc-all-langpack` is optimized to again use
the locale cache (locale-archive) to accelerate lookups for those
users who prefer to have all languages installed. Previously we
had `glibc-all-langpack` as a meta-package of all the other language
packs, but that is no longer the case.

(3) Provide glibc-minimal-langpack

We add a minimal language pack that provides no languages. This
allows you to install glibc-minimal-langpack and then uninstall
glibc-all-langpacks and operate entirely under C, POSIX, and C.UTF-8
locales.

This should resolve all problems that users have been seeing with
the upgrade.

Thank you for your patience. We really messed up here. In the future
we'll make sure we cover all the upgrade cases more carefully.

(4) Remove all languages to minimize installed size

You can still minimize your installed size, but it's slighltly
harder now. First you must install a valid language pack, and then
remove `glibc-all-langpacks` to reduce the size of installed languages.

Cheers,
Carlos.

[Remi Collet]

Post by Carlos O'Donell
The glibc in Fedora rawhide and F24 is now split by
language packs. We have over 180 supported languages
in glibc, and those have been split into langpacks
for transparent install and support via dnf.

I may understand the need.

But what about "server" ?

If you install a webapp which use gettext, the locale will be set
according to the conneected user (ex Horde)

Does that mean that such webapp should requires "qlibc-all-langpacks" ?


Remi.

[Matthew Miller]

Post by Remi Collet
I may understand the need.
But what about "server" ?
If you install a webapp which use gettext, the locale will be set
according to the conneected user (ex Horde)
Does that mean that such webapp should requires "qlibc-all-langpacks" ?

I'm not sure of the overall answer, but definitely please don't make
it a _hard_ Requires — use Suggests or Recommends instead.