June 03 2021

Retiring the multilib project

Michał Górny (mgorny) June 03, 2021, 20:41

I created the Multilib project back in November 2013 (though the effort itself started roughly a year earlier) with the goal of maintaining the multilib eclasses and porting Gentoo packages to them. Back in the day, we were even requested to co-maintain a few packages whose maintainers were opposed to multilib ports. In June 2015, last of the emul-linux-x86 packages were removed and our work has concluded.

The project continued to exist for the purpose of maintaining the eclasses and providing advice. Today, I can say that the project has served its purpose and it is time to retire it. Most of the team members have already left, the multilib knowledge that we advised on before is now common developer knowledge. I am planning to take care of the project-maintained eclasses personally, and move the relevant documentation to the general wiki space.

At the same time, I would like to take this opportunity to tell the history of our little multilib project.

Gentoo before gx86-multilib

In the old days, the multilib as seen by the majority of Gentoo users consisted of two components: multilib toolchain packages and emul-linux-x86 packages.

The toolchain multilib support exists pretty much in its original form to this day. It consists of a multilib USE flag and an ABI list stored in the selected profile. The rough idea is that bootstrapping a toolchain with a superset of its current ABIs is non-trivial, so the users generally choose a particular multilib or non-multilib variant when installing Gentoo, and do not change it afterwards. The multilib project didn’t really touch this part.

The emul-linux-x86 packages were specifically focused on non-toolchain packages. Back in the day, they consisted of a few sets of precompiled 32-bit libraries for amd64. If you needed to run a proprietary 32-bit app or compile wine, they had to depend on a few sets, e.g.:

amd64? (
    app-emulation/emul-linux-x86-xlibs
    app-emulation/emul-linux-x86-soundlibs
)

The sets generally included the current stable versions of packages and were rebuilt every few months.

Simultaneously, an alternative to this solution was developed (and is being developed to this day): multilib-portage, a Portage fork that was designed specifically to build all packages for multiple ABIs. Unlike the other solutions, multilib-portage minimized development effort and worked on top of regular Gentoo packages. However, it never reached production readiness.

The gx86-multilib design

The gx86-multilib effort was intended to provide a multilib solution entirely within the scope of the Gentoo repository (still named gentoo-x86 at the time, hence the name), i.e. without having to modify EAPI or package managers. It was supposed to be something between emul-linux-x86 and multilib-portage, building all non-native libraries from source but requiring explicit support from packages.

It only seemed natural to utilize USE_EXPAND the same way as PYTHON_TARGETS did for Python. At the same time, splitting ABIs per architecture made it possible to use USE_EXPAND_HIDDEN to hide irrelevant flags from users. So e.g. amd64 multilib users see only ABI_X86, PPC64 users see ABI_PPC and so on.

The default ABI for a given platform is always forced on. This made it possible to keep things working for non-multilib packages without adding any multilib awareness to them, and at the same time cleanly handle profiles that do not do multilib at all. Multilib packages use ${MULTILIB_USEDEP} to enforce ABI match on their multilib depdencies; non-multilib packages just use plain deps and can expect the native ABI to be always enabled.

Eclasses were a natural place to implement all this logic. In the end, they formed a hierarchical structure. The pre-existing multilib.eclass already provided a few low-level functions needed to set up multilib builds. On top of it, multilib-build.eclass was created that provided low-level functions specific to the gx86-multilib — handling USE flags, running the builds and some specific helper functions. On top of it, high-level sub-phase-based multilib-minimal.eclass was created that made writing generic ebuilds easy. Then, on top of that the specific autotools-multilib.eclass and cmake-multilib.eclass existed.

Historically, the order was a little different. autotools-multilib.eclass came first. Then, the common logic was split into multilib-build.eclass and cmake-multilib.eclass came to be. Finally, multilib-minimal.eclass was introduced and a few months later the other eclasses started reusing it.

The reception and porting efforts

The eclasses had a mixed reception. They followed my philosophy of getting things done, today. This disagreed with purists who believed we should look for a better solution. Many of the developers believed that multilib-portage was the way forward (after all, it did not require changing ebuilds), though they did not seem to be that much concerned about having a clear plan of action. When I’ve pointed out that things need to be formally specified, the answer was roughly to dump whatever’s in multilib-portage at the time into the spec. As you can guess, no spec was ever written.

Nevertheless, porting ebuilds to the new framework proceeded over time. In some cases, we had to deal with varying level of opposition. In the most extreme cases, we had to work out a compromise and become co-maintainers of these packages in order to provide direct support for any port-related problems. However, as time went by more people joined the cause, and today it is pretty natural that maintainers add multilib support themselves. In fact, I believe that things went a bit out of control, as multilib is being added to packages where it is not really needed.

In its early years, gx86-multilib had to coexist with the older emul-linux-x86 packages. Since both groups of packages installed the same files, collisions were inevitable. Every few converted packages, we had to revbump the respective emul-linux-x86 sets dropping the colliding libraries. Later on, we had to start replacing old dependencies on emul-linux-x86 packages (now metapackages) with the actual lists of needed libraries. Naturally, this meant that someone actually had to figure out what the dependencies were — often for fetch-restricted packages that we simply didn’t have distfiles for.

In the end, everything went fine. All relevant packages were ported, emul-linux-x86 sets were retired. The team stayed around for a few years, updating the eclasses as need be. Many new packages gained multilib support even though it wasn’t strictly needed for anything. Multilib-foo became common knowledge.

The future

Our multilib effort is still alive and kicking. At the very least, it serves as the foundation for 32-bit Wine. While the Multilib project itself has been disbanded, its legacy lives on and it is not likely to become obsolete anytime soon. From a quick grep, there are around 600 multilib-enabled packages in ::gentoo at the moment and it is quite likely that there will be more.

The multilib-portage project is still being developed but it does not seem likely to be able to escape its niche. The eclass approach is easier, more portable and more efficient. You don’t have to modify the package manager, you don’t have to build everything multiple times; ideally, you only build library parts for all ABIs.

Support for multilib on non-x86 platforms is an open question. After all, the whole multilib effort was primarily focused on providing compatibility with old 32-bit executables on x86. While some platforms technically can provide multilib, it is not clear how much of that is actually useful to the users, and how much is a cargo cult. Support for additional targets has historically proven troublesome by causing exponential explosion of USE flags.

Some people were proposing switching to Debian-style multiarch layout (e.g. /usr/lib/x86_64-linux-gnu instead of /usr/lib64). However, I have never seen a strong reason to do that. After all, traditional libdirs are well-defined in the ABI specifications while multiarch is a custom Debian invention. In the end, it would be about moving things around and then patching packages into supporting non-standard locations. It would go against one of the primary Gentoo principles of providing a vanilla development environment. And that only shortly after we’ve finally gotten rid of the custom /usr/lib32 in favor of backwards-compatible /usr/lib.

So, while the Multilib project has been retired now, multilib itself is all but dead. We still use it, we still need it and we will probably still work on it in the future.

I created the Multilib project back in November 2013 (though the effort itself started roughly a year earlier) with the goal of maintaining the multilib eclasses and porting Gentoo packages to them. Back in the day, we were even requested to co-maintain a few packages whose maintainers were opposed to multilib ports. In June 2015, last of the emul-linux-x86 packages were removed and our work has concluded.

The project continued to exist for the purpose of maintaining the eclasses and providing advice. Today, I can say that the project has served its purpose and it is time to retire it. Most of the team members have already left, the multilib knowledge that we advised on before is now common developer knowledge. I am planning to take care of the project-maintained eclasses personally, and move the relevant documentation to the general wiki space.

At the same time, I would like to take this opportunity to tell the history of our little multilib project.

Gentoo before gx86-multilib

In the old days, the multilib as seen by the majority of Gentoo users consisted of two components: multilib toolchain packages and emul-linux-x86 packages.

The toolchain multilib support exists pretty much in its original form to this day. It consists of a multilib USE flag and an ABI list stored in the selected profile. The rough idea is that bootstrapping a toolchain with a superset of its current ABIs is non-trivial, so the users generally choose a particular multilib or non-multilib variant when installing Gentoo, and do not change it afterwards. The multilib project didn’t really touch this part.

The emul-linux-x86 packages were specifically focused on non-toolchain packages. Back in the day, they consisted of a few sets of precompiled 32-bit libraries for amd64. If you needed to run a proprietary 32-bit app or compile wine, they had to depend on a few sets, e.g.:

amd64? (
    app-emulation/emul-linux-x86-xlibs
    app-emulation/emul-linux-x86-soundlibs
)

The sets generally included the current stable versions of packages and were rebuilt every few months.

Simultaneously, an alternative to this solution was developed (and is being developed to this day): multilib-portage, a Portage fork that was designed specifically to build all packages for multiple ABIs. Unlike the other solutions, multilib-portage minimized development effort and worked on top of regular Gentoo packages. However, it never reached production readiness.

The gx86-multilib design

The gx86-multilib effort was intended to provide a multilib solution entirely within the scope of the Gentoo repository (still named gentoo-x86 at the time, hence the name), i.e. without having to modify EAPI or package managers. It was supposed to be something between emul-linux-x86 and multilib-portage, building all non-native libraries from source but requiring explicit support from packages.

It only seemed natural to utilize USE_EXPAND the same way as PYTHON_TARGETS did for Python. At the same time, splitting ABIs per architecture made it possible to use USE_EXPAND_HIDDEN to hide irrelevant flags from users. So e.g. amd64 multilib users see only ABI_X86, PPC64 users see ABI_PPC and so on.

The default ABI for a given platform is always forced on. This made it possible to keep things working for non-multilib packages without adding any multilib awareness to them, and at the same time cleanly handle profiles that do not do multilib at all. Multilib packages use ${MULTILIB_USEDEP} to enforce ABI match on their multilib depdencies; non-multilib packages just use plain deps and can expect the native ABI to be always enabled.

Eclasses were a natural place to implement all this logic. In the end, they formed a hierarchical structure. The pre-existing multilib.eclass already provided a few low-level functions needed to set up multilib builds. On top of it, multilib-build.eclass was created that provided low-level functions specific to the gx86-multilib — handling USE flags, running the builds and some specific helper functions. On top of it, high-level sub-phase-based multilib-minimal.eclass was created that made writing generic ebuilds easy. Then, on top of that the specific autotools-multilib.eclass and cmake-multilib.eclass existed.

Historically, the order was a little different. autotools-multilib.eclass came first. Then, the common logic was split into multilib-build.eclass and cmake-multilib.eclass came to be. Finally, multilib-minimal.eclass was introduced and a few months later the other eclasses started reusing it.

The reception and porting efforts

The eclasses had a mixed reception. They followed my philosophy of getting things done, today. This disagreed with purists who believed we should look for a better solution. Many of the developers believed that multilib-portage was the way forward (after all, it did not require changing ebuilds), though they did not seem to be that much concerned about having a clear plan of action. When I’ve pointed out that things need to be formally specified, the answer was roughly to dump whatever’s in multilib-portage at the time into the spec. As you can guess, no spec was ever written.

Nevertheless, porting ebuilds to the new framework proceeded over time. In some cases, we had to deal with varying level of opposition. In the most extreme cases, we had to work out a compromise and become co-maintainers of these packages in order to provide direct support for any port-related problems. However, as time went by more people joined the cause, and today it is pretty natural that maintainers add multilib support themselves. In fact, I believe that things went a bit out of control, as multilib is being added to packages where it is not really needed.

In its early years, gx86-multilib had to coexist with the older emul-linux-x86 packages. Since both groups of packages installed the same files, collisions were inevitable. Every few converted packages, we had to revbump the respective emul-linux-x86 sets dropping the colliding libraries. Later on, we had to start replacing old dependencies on emul-linux-x86 packages (now metapackages) with the actual lists of needed libraries. Naturally, this meant that someone actually had to figure out what the dependencies were — often for fetch-restricted packages that we simply didn’t have distfiles for.

In the end, everything went fine. All relevant packages were ported, emul-linux-x86 sets were retired. The team stayed around for a few years, updating the eclasses as need be. Many new packages gained multilib support even though it wasn’t strictly needed for anything. Multilib-foo became common knowledge.

The future

Our multilib effort is still alive and kicking. At the very least, it serves as the foundation for 32-bit Wine. While the Multilib project itself has been disbanded, its legacy lives on and it is not likely to become obsolete anytime soon. From a quick grep, there are around 600 multilib-enabled packages in ::gentoo at the moment and it is quite likely that there will be more.

The multilib-portage project is still being developed but it does not seem likely to be able to escape its niche. The eclass approach is easier, more portable and more efficient. You don’t have to modify the package manager, you don’t have to build everything multiple times; ideally, you only build library parts for all ABIs.

Support for multilib on non-x86 platforms is an open question. After all, the whole multilib effort was primarily focused on providing compatibility with old 32-bit executables on x86. While some platforms technically can provide multilib, it is not clear how much of that is actually useful to the users, and how much is a cargo cult. Support for additional targets has historically proven troublesome by causing exponential explosion of USE flags.

Some people were proposing switching to Debian-style multiarch layout (e.g. /usr/lib/x86_64-linux-gnu instead of /usr/lib64). However, I have never seen a strong reason to do that. After all, traditional libdirs are well-defined in the ABI specifications while multiarch is a custom Debian invention. In the end, it would be about moving things around and then patching packages into supporting non-standard locations. It would go against one of the primary Gentoo principles of providing a vanilla development environment. And that only shortly after we’ve finally gotten rid of the custom /usr/lib32 in favor of backwards-compatible /usr/lib.

So, while the Multilib project has been retired now, multilib itself is all but dead. We still use it, we still need it and we will probably still work on it in the future.

May 26 2021

Gentoo Freenode channels have been hijacked

Gentoo News (GentooNews) May 26, 2021, 5:00

Today (2021-05-26) a large number of Gentoo channels have been hijacked by Freenode staff, including channels that were not yet migrated to Libera.chat. We cannot perceive this otherwise than as an open act of hostility and we have effectively left Freenode.

Please note that at this point the only official Gentoo IRC channels, as well as developer accounts, can be found on Libera Chat.

Today (2021-05-26) a large number of Gentoo channels have been hijacked by Freenode staff, including channels that were not yet migrated to Libera.chat. We cannot perceive this otherwise than as an open act of hostility and we have effectively left Freenode.

Please note that at this point the only official Gentoo IRC channels, as well as developer accounts, can be found on Libera Chat.

Gentoo IRC presence moving to Libera Chat

Gentoo News (GentooNews) May 23, 2021, 5:00

The Gentoo Council held an emergency single agenda item meeting today. At this meeting, we have decided to move the official IRC presence of Gentoo to the Libera Chat IRC network. We intend to have this move complete at latest by 13/June/2021. A full log of the meeting will be available for download soon.

At the moment it is unclear whether we will retain any presence on Freenode at all; we urge all users of the #gentoo channel namespace to move to Libera Chat immediately. IRC channel names will (mostly) remain identical. You will be able to recognize Gentoo developers on Libera Chat by their IRC cloak in the usual form gentoo/developer/*. All other technical aspects will feel rather familiar to all of us as well. Detailed instructions for setting up various IRC clients can be found on the help pages of the IRC network.

Libera.Chat logo

The Gentoo Council held an emergency single agenda item meeting today. At this meeting, we have decided to move the official IRC presence of Gentoo to the Libera Chat IRC network. We intend to have this move complete at latest by 13/June/2021. A full log of the meeting will be available for download soon.

At the moment it is unclear whether we will retain any presence on Freenode at all; we urge all users of the #gentoo channel namespace to move to Libera Chat immediately. IRC channel names will (mostly) remain identical. You will be able to recognize Gentoo developers on Libera Chat by their IRC cloak in the usual form gentoo/developer/*. All other technical aspects will feel rather familiar to all of us as well. Detailed instructions for setting up various IRC clients can be found on the help pages of the IRC network.

May 21 2021

From build-dir to venv — testing Python packages in Gentoo

Michał Górny (mgorny) May 21, 2021, 19:40

A lot of Python packages assume that their tests will be run after installing the package. This is quite a reasonable assumption if you take that the tests are primarily run in dedicated testing environments such as CI deployments or test runners such as tox. However, this does not necessarily fit the Gentoo packaging model where packages are installed system-wide, and the tests are run between compile and install phases.

In great many cases, things work out of the box (because the modules are found relatively to the current directory), or require only minimal PYTHONPATH adjustments. In others, we found it necessary to put a varying amount of effort to create a local installation of the package that is suitable for testing.

In this post, I would like to shortly explore the various solutions to the problem we’ve used over the years, from simple uses of build directory to the newest ideas based on virtual environments.

Testing against the build directory

As I have indicated above, great many packages work just fine with the correct PYTHONPATH setting. However, not all packages provide ready-to-use source trees and even if they do, there’s the matter of either having to manually specify the path to them or have more or less reliable automation guess it. Fortunately, there’s a simple solution.

The traditional distutils/setuptools build process consists of two phases: the build phase and the install phase. The build phase is primarily about copying the files from their respective source directories to a unified package tree in a build directory, while the install phase is generally about installing the files found in the build directory. Besides just reintegrating sources, the build phase may also involve other important taks: compiling the extensions written in C or converting sources from Python 2 to Python 3 (which is becoming rare). Given that the build command is run in src_compile, this makes the build directory a good candidate for use in tests.

This is precisely what distutils-r1.eclass does out of the box. It ensures that the build commands write to a predictable location, and it adds that location to PYTHONPATH. This ensures that the just-built package is found by Python when trying to import its modules. That is — unless the package residing in the current directory takes precedence. In either case, it means that most of the time things just work, and sometimes just have to restort to simple hacks such as changing the current directory.

distutils_install_for_testing (home layout)

While the build directory method worked for many packages, it had its limitation. To list a few I can think of:

  • Script wrappers for entry points were not created (and even regular scripts were not added to PATH due to a historical mistake), so tests that relied on being able to call installed executables did not work.
  • Package metadata (.egg-info) was not included, so pkg_resources (and now the more modern importlib.metadata) modules may have had trouble finding the package.
  • Namespace packages were not handled properly.

The last point was the deal breaker here. Remember that we’re talking of the times when Python 2.7 was still widely supported. If we were testing a zope.foo package that happened to depend on zope.bar, then we were in trouble. The top-level zope package that we’ve just added to PYTHONPATH had only the foo submodule but bar had to be gotten from system site-packages!

Back in the day, I did not know much about the internals of these things. I was looking for an easy working solution, and I have found one. I have discovered that using setup.py install --home=... (vs setup.py install --root=... that we used to install into D) happened to install a layout that made namespaces just work! This was just great!

This how the original implementation of distutils_install_for_testing came around. The rough idea was to put this –home install layout on PYTHONPATH and reap all the benefits of having the package installed before running tests.

Root layout

The original dift layout was good while it worked. But then it stopped. I don’t know the exact version of setuptools or the exact change but the magic just stopped working. Good news is that it was just a few months ago, and we were already deep in removing Python 2.7, so we did not have to worry about namespaces that much (namespaces are much easier in Python 3 as they work via empty directories without special magic).

The simplest solution I could think of was to stop relying on the home layout, and instead use the same root layout as used for our regular installs. This did not include as much magic but solved the important problems nevertheless. Entry point wrappers were installed, namespaces worked of their own accord most of the time.

I’ve added a new --via-root parameter to change dift mode, and --via-home to force the old behavior. By the end of January, I have flipped the default and we were happily using the new layout since then. Except that it didn’t really solve all the problems.

Virtualenv layout

The biggest limitations of the both dift layouts is that they’ve relied on PYTHONPATH. However, not everything in the Python world respects path overrides. To list just two examples: the test suite of werkzeug relies on overwriting PYTHONPATH for spawned processes, and tox fails to find its own installed package.

I have tried various hacks to resolve this, to no avail. The solution that somewhat worked was to require the package to be actually installed before running the tests but that was really inconvenient. Interestingly enough, virtualenvs rely on some internal Python magic to actually override module search path without relying on PYTHONPATH.

The most recent dift --via-venv variant that I’ve just submitted for mailing list review uses exactly this. That is, it uses the built-in Python 3 venv module (not to be confused with the third-party virtualenv).

Now, normally a virtualenv creates an isolated environment where all dependencies have to be installed explicitly. However, there is a --system-site-packages option that avoids this. The packages installed inside the virtualenv (i.e. the tested package) will take precedence but other packages will be imported from the system site-packages directory. That’s just what we need!

I have so far tested this new method on two problematic packages (werkzeug and tox). It might be just the thing that resolves all the problems that were previously resolved via the home layout. Or it might not. I do not know yet whether we’ll be switching default again. Time will tell.

A lot of Python packages assume that their tests will be run after installing the package. This is quite a reasonable assumption if you take that the tests are primarily run in dedicated testing environments such as CI deployments or test runners such as tox. However, this does not necessarily fit the Gentoo packaging model where packages are installed system-wide, and the tests are run between compile and install phases.

In great many cases, things work out of the box (because the modules are found relatively to the current directory), or require only minimal PYTHONPATH adjustments. In others, we found it necessary to put a varying amount of effort to create a local installation of the package that is suitable for testing.

In this post, I would like to shortly explore the various solutions to the problem we’ve used over the years, from simple uses of build directory to the newest ideas based on virtual environments.

Testing against the build directory

As I have indicated above, great many packages work just fine with the correct PYTHONPATH setting. However, not all packages provide ready-to-use source trees and even if they do, there’s the matter of either having to manually specify the path to them or have more or less reliable automation guess it. Fortunately, there’s a simple solution.

The traditional distutils/setuptools build process consists of two phases: the build phase and the install phase. The build phase is primarily about copying the files from their respective source directories to a unified package tree in a build directory, while the install phase is generally about installing the files found in the build directory. Besides just reintegrating sources, the build phase may also involve other important taks: compiling the extensions written in C or converting sources from Python 2 to Python 3 (which is becoming rare). Given that the build command is run in src_compile, this makes the build directory a good candidate for use in tests.

This is precisely what distutils-r1.eclass does out of the box. It ensures that the build commands write to a predictable location, and it adds that location to PYTHONPATH. This ensures that the just-built package is found by Python when trying to import its modules. That is — unless the package residing in the current directory takes precedence. In either case, it means that most of the time things just work, and sometimes just have to restort to simple hacks such as changing the current directory.

distutils_install_for_testing (home layout)

While the build directory method worked for many packages, it had its limitation. To list a few I can think of:

  • Script wrappers for entry points were not created (and even regular scripts were not added to PATH due to a historical mistake), so tests that relied on being able to call installed executables did not work.
  • Package metadata (.egg-info) was not included, so pkg_resources (and now the more modern importlib.metadata) modules may have had trouble finding the package.
  • Namespace packages were not handled properly.

The last point was the deal breaker here. Remember that we’re talking of the times when Python 2.7 was still widely supported. If we were testing a zope.foo package that happened to depend on zope.bar, then we were in trouble. The top-level zope package that we’ve just added to PYTHONPATH had only the foo submodule but bar had to be gotten from system site-packages!

Back in the day, I did not know much about the internals of these things. I was looking for an easy working solution, and I have found one. I have discovered that using setup.py install --home=... (vs setup.py install --root=... that we used to install into D) happened to install a layout that made namespaces just work! This was just great!

This how the original implementation of distutils_install_for_testing came around. The rough idea was to put this –home install layout on PYTHONPATH and reap all the benefits of having the package installed before running tests.

Root layout

The original dift layout was good while it worked. But then it stopped. I don’t know the exact version of setuptools or the exact change but the magic just stopped working. Good news is that it was just a few months ago, and we were already deep in removing Python 2.7, so we did not have to worry about namespaces that much (namespaces are much easier in Python 3 as they work via empty directories without special magic).

The simplest solution I could think of was to stop relying on the home layout, and instead use the same root layout as used for our regular installs. This did not include as much magic but solved the important problems nevertheless. Entry point wrappers were installed, namespaces worked of their own accord most of the time.

I’ve added a new --via-root parameter to change dift mode, and --via-home to force the old behavior. By the end of January, I have flipped the default and we were happily using the new layout since then. Except that it didn’t really solve all the problems.

Virtualenv layout

The biggest limitations of the both dift layouts is that they’ve relied on PYTHONPATH. However, not everything in the Python world respects path overrides. To list just two examples: the test suite of werkzeug relies on overwriting PYTHONPATH for spawned processes, and tox fails to find its own installed package.

I have tried various hacks to resolve this, to no avail. The solution that somewhat worked was to require the package to be actually installed before running the tests but that was really inconvenient. Interestingly enough, virtualenvs rely on some internal Python magic to actually override module search path without relying on PYTHONPATH.

The most recent dift --via-venv variant that I’ve just submitted for mailing list review uses exactly this. That is, it uses the built-in Python 3 venv module (not to be confused with the third-party virtualenv).

Now, normally a virtualenv creates an isolated environment where all dependencies have to be installed explicitly. However, there is a --system-site-packages option that avoids this. The packages installed inside the virtualenv (i.e. the tested package) will take precedence but other packages will be imported from the system site-packages directory. That’s just what we need!

I have so far tested this new method on two problematic packages (werkzeug and tox). It might be just the thing that resolves all the problems that were previously resolved via the home layout. Or it might not. I do not know yet whether we’ll be switching default again. Time will tell.

May 20 2021

Freenode IRC and Gentoo

Gentoo News (GentooNews) May 20, 2021, 5:00

According to the information published recently, there have been major changes in the way the Freenode IRC network is administered. This has resulted in a number of staff members raising concerns about the new administration and/or resigning. A large number of open source projects have already announced the transition to other IRC networks, or are actively discussing it.

It is not yet clear whether and how these changes will affect Gentoo. We are observing as the situation develops. It is possible that we will decide to move the official Gentoo channels to another network in the best interest of our users. At the same time, we realize that such a move will be an inconvenience to them.

At the same time, it has came to our attention that certain individuals have been using the situation to impersonate Gentoo developers on other IRC networks. The official Gentoo developers can be identified on Freenode by their gentoo/developer cloak. If we move to another network, we will announce claiming a respective cloak.

Please check this page for future updates.

More information on the Freenode situation can be found at:

  • Christian (Fuchs)’s Freenode resignation
  • @freenodestaff tweet
  • Open Letter On freenode’s independence
  • Andrew Lee, We grew up with IRC. Let’s take it further.
2021-05-22 update

The Gentoo Council will be meeting tomorrow (Sunday, 2021-05-23) at 19:00 UTC to discuss the problem and the possible solutions.

The Gentoo Group Contacts team has been taking steps in order to ensure readiness for the most likely options.

According to the information published recently, there have been major changes in the way the Freenode IRC network is administered. This has resulted in a number of staff members raising concerns about the new administration and/or resigning. A large number of open source projects have already announced the transition to other IRC networks, or are actively discussing it.

It is not yet clear whether and how these changes will affect Gentoo. We are observing as the situation develops. It is possible that we will decide to move the official Gentoo channels to another network in the best interest of our users. At the same time, we realize that such a move will be an inconvenience to them.

At the same time, it has came to our attention that certain individuals have been using the situation to impersonate Gentoo developers on other IRC networks. The official Gentoo developers can be identified on Freenode by their gentoo/developer cloak. If we move to another network, we will announce claiming a respective cloak.

Please check this page for future updates.

More information on the Freenode situation can be found at:

2021-05-22 update

The Gentoo Council will be meeting tomorrow (Sunday, 2021-05-23) at 19:00 UTC to discuss the problem and the possible solutions.

The Gentoo Group Contacts team has been taking steps in order to ensure readiness for the most likely options.

May 18 2021

Google Summer of Code 2021 students welcome

Gentoo News (GentooNews) May 18, 2021, 5:00

We are glad to welcome Leo and Mark to the Google Summer of Code 2021.

Mark will work on improving Catalyst, our release building tool. Leo will work on improving our Java packaging support, with a special focus on big-data and scientific software.

We are glad to welcome Leo and Mark to the Google Summer of Code 2021.

Mark will work on improving Catalyst, our release building tool. Leo will work on improving our Java packaging support, with a special focus on big-data and scientific software.

May 06 2021

10 Years’ Perspective on Python in Gentoo

Michał Górny (mgorny) May 06, 2021, 7:55

I’m a Gentoo developer for over 10 years already. I’ve been doing a lot of different things throughout that period. However, Python was pretty much always somewhere within my area of interest. I don’t really recall how it all started. Maybe it had something to do with Portage being written in Python. Maybe it was the natural next step after programming in Perl.

I feel like the upcoming switch to Python 3.9 is the last step in the prolonged effort of catching up with Python. Over the last years, we’ve been working real hard to move Python support forward, to bump neglected packages, to enable testing where tests are available, to test packages on new targets and unmask new targets as soon as possible. We have improved the processes a lot. Back when we were switching to Python 3.4, it took almost a year from the first false start attempt to the actual change. We started using Python 3.5 by default after upstream dropped bugfix support for it. In a month from now, we are going to start using Python 3.9 even before 3.10 final is released.

I think this is a great opportunity to look back and see what changed in the Gentoo Python ecosystem, in the last 10 years.

Python package ebuilds 10 years ago

Do you know how a Python package ebuild looked like 10 years ago? Let’s take gentoopm-0.1 as an example (reformatted to fit the narrow layout better):


# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.1.ebuild,v 1.1 2011/07/15 19:05:26 mgorny Exp $

EAPI=3

PYTHON_DEPEND='*:2.6'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS='2.4 2.5'
DISTUTILS_SRC_TEST=setup.py

inherit base distutils

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="github.com/gentoopm/"
SRC_URI="cloud.github.com/downloads/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.8.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="dev-python/epydoc"
PDEPEND="app-admin/eselect-package-manager"

src_prepare() {
  base_src_prepare
  distutils_src_prepare
}

src_compile() {
  distutils_src_compile

  if use doc; then
    "$(PYTHON -2)" setup.py doc || die
  fi
}

src_install() {
  distutils_src_install

  if use doc; then
    dohtml -r doc/* || die
  fi
}

This ebuild is actually using the newer API of python.eclass that is enabled via SUPPORT_PYTHON_ABIS. It provides support for installing for multiple implementations (like the modern python-r1 eclass). PYTHON_DEPEND is used to control the dependency string added to ebuild. The magical syntax here means that the ebuild supports both Python 2 and Python 3, from Python 2.6 upwards. RESTRICT_PYTHON_ABIS opts out support for Python versions prior to 2.6. Note the redundancy — PYTHON_DEPEND controls the dependency, specified as a range of Python 2 and/or Python 3 versions, RESTRICT_PYTHON_ABIS controls versions used at build time and needs to explicitly exclude all unsupported branches.

Back then, there were no PYTHON_TARGETS to control what was built. Instead, the eclass defaulted to using whatever was selected via eselect python, with the option to override it via setting USE_PYTHON in make.conf. Therefore, there were no cross-package USE dependencies and you had to run python-updater to verify whether all packages are built for the current interpreter, and rebuild these that were not.

Still, support for multiple ABIs, as the eclass called different branches/implementations of Python, was a major step forward. It was added around the time that the first releases of Python 3 were published, and our users have been relying on it to support a combination of Python 2 and Python 3 for years. Today, we’re primarily using it to aid developers in testing their packages and to provide a safer upgrade experience.

The python.eclass stalemate

Unfortunately, things at the time were not all that great. Late 2010 marks a conflict between the primary Python developer and the rest of the community, primarily due to the major breakage being caused by the changes in Python support. By mid-2011, it was pretty clear that there is no chance to resolve the conflict. The in-Gentoo version of python.eclass was failing to get EAPI 4 support for 6 months already, while an incompatible version continued being developed in the (old) Python overlay. As Dirkjan Ochtman related in his mail:

I guess by now pretty much everyone knows that the python eclass is rather complex, and that this poses some problems. This has also been an important cause for the disagreements between Arfrever and some of the other developers. Since it appears that Arfrever won’t be committing much code to gentoo-x86 in the near future, I’m trying to figure out where we should go with the python.eclass. […]

Dirkjan Ochtman, 2011-06-27, [gentoo-dev] The Python problem

Eventually, some of the changes from the Python overlay were backported to the eclass and EAPI 4 support was added. Nevertheless, at this point it was pretty clear that we need a new way forward. Unfortunately, the discussions were leading nowhere. With the primary eclass maintainer retired, nobody really comprehended most of the eclass, nor were able to afford the time to figure it out. At the same time, involved parties wanted to preserve backwards compatibility while moving forward.

The tie breaker: python-distutils-ng

Some of you might find it surprising that PYTHON_TARGETS are not really a python-r1 invention. Back in March 2012, when Python team was still unable to find a way forward with python.eclass, Krzysztof Pawlik (nelchael) has committed a new python-distutils-ng.eclass. It has never grown popular, and it has been replaced by the python-r1 suite before it ever started being a meaningful replacement for python.eclass. Still, it served an important impulse that made what came after possible.

Here’s a newer gentoopm ebuild using the new eclass (again reformatted):


# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.2.5-r1.ebuild,v 1.1 2012/05/26 10:11:21 mgorny Exp $

EAPI=4
PYTHON_COMPAT='python2_6 python2_7 python3_1 python3_2'

inherit base python-distutils-ng

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="github.com/mgorny/gentoopm/"
SRC_URI="mirror://github/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~mips ~x86 ~x86-fbsd"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.10.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="doc? ( dev-python/epydoc )"
PDEPEND="app-admin/eselect-package-manager"

python_prepare_all() {
  base_src_prepare
}

src_compile() {
  python-distutils-ng_src_compile
  if use doc; then
    "${PYTHON}" setup.py doc || die
  fi
}

python_install_all() {
  if use doc; then
    dohtml -r doc/*
  fi
}

Just looking at the code, you may see that python-r1 has inherited a lot after this eclass. python-distutils-ng in turn followed some of the good practices introduced before in the ruby-ng eclass. It introduced PYTHON_TARGETS to provide explicit visible control over implementations used for the build — though notably it did not include a way for packages to depend on matching flags (i.e. the equivalent of ${PYTHON_USEDEP}). It also used the sub-phase approach that makes distutils-r1 and ruby-ng eclasses much more convenient than the traditional python.eclass approach that roughly resembled using python_foreach_impl all the time.

What’s really important is that python-distutils-ng carved a way forward. It’s been a great inspiration and a proof of concept. It has shown that we do not have to preserve compatibility with python.eclass forever, or have to learn its inner workings before starting to solve problems. I can’t say how Python would look today if it did not happen but I can say with certainly that python-r1 would not happen so soon if it weren’t for it.

python-r1

In October 2012, the first version of python-r1 was committed. It combined some of the very good ideas of python-distutils-ng with some of my own.  The goal was not to provide an immediate replacement for python.eclass. Instead, the plan was to start simple and add new features as they turned out to be necessary. Not everything went perfectly but I dare say that the design has stood the test of time. While I feel like the eclasses ended up being more complex than I wished they would be, they still work fine with no replacement in sight and they serve as inspiration to other eclasses.

For completeness, here’s a 2017 gentoopm live ebuild that uses a pretty complete distutils-r1 feature set:


# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6
PYTHON_COMPAT=( python{2_7,3_4,3_5,3_6} pypy )

EGIT_REPO_URI="github.com/mgorny/gentoopm.git"
inherit distutils-r1 git-r3

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="github.com/mgorny/gentoopm/"
SRC_URI=""

LICENSE="BSD-2"
SLOT="0"
KEYWORDS=""
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/pkgcore-0.9.4[${PYTHON_USEDEP}]
    >=sys-apps/portage-2.1.10.3[${PYTHON_USEDEP}]
    >=sys-apps/paludis-3.0.0_pre20170219[python,${PYTHON_USEDEP}]
  )"
DEPEND="
  doc? (
    dev-python/epydoc[$(python_gen_usedep python2_7)]
  )"
PDEPEND="app-eselect/eselect-package-manager"

REQUIRED_USE="
  doc? ( $(python_gen_useflags python2_7) )"

src_configure() {
  use doc && DISTUTILS_ALL_SUBPHASE_IMPLS=( python2.7 )
  distutils-r1_src_configure
}

python_compile_all() {
  use doc && esetup.py doc
}

python_test() {
  esetup.py test
}

python_install_all() {
  use doc && local HTML_DOCS=( doc/. )
  distutils-r1_python_install_all
}

The new eclasses have employed the same PYTHON_TARGETS flags for the general implementation choice but also added PYTHON_SINGLE_TARGET to make choosing the implementation more convenient (and predictable at the same time) when the package did not permit choosing more than one. The distutils-r1 eclass reused the great idea of sub-phases to make partial alterations to the phases easier.

The unique ideas included:

  • the split into more eclasses by functionality (vs switching the mode via variables as done in python.eclass)
  • exposing dependencies and REQUIRED_USE constraints via variables to be used in ebuild instead of writing elaborate mechanisms for adding dependencies
  • avoiding command substitution in global scope to keep metadata regeneration fast

The eclasses evolved a lot over the years. Some of the original ideas turned out pretty bad, e.g. trying to run sub-phases in parallel (which broke a lot of stuff for minor performance gain) or the horribly complex original interaction between PYTHON_TARGETS and PYTHON_SINGLE_TARGET. Ebuilds often missed dependencies and REQUIRED_USE constraints, until we finally made the pkgcheck-based CI report that.

The migration to new eclasses took many years. Initially, python-r1 ebuilds were even depending on python.eclass ebuilds. The old eclass was not removed until March 2017, i.e. 4.5 years after introducing the new eclasses.

Testing on new Python targets

We went for the opt-in approach with PYTHON_COMPAT. This means that for every new Python target added, we start with no packages supporting it and have to iterate over all of the packages adding the support. It’s a lot of work and it has repeatedly caused users pain due to packages not being ported in time for the big switch to the next version. Some people have complained about that and suggested that we should go for opt-out instead. However, if you think about it, opt-in is the only way to go.

The big deal is that for any particular package to support a new implementation, all of its Python dependencies need to support it as well. With the opt-in approach, it means that we’re doing the testing dependency-first, and reaching the bigger packages only when we confirm that there’s at least a single version of every dependency that works for it. If we do things right, users don’t even see any regressions.

If we went for the opt-out approach, all packages would suddenly claim to support the new version. Now, this wouldn’t be that bad if we were actually able to do a big CI run for all packages — but we can’t since a lot of them do not have proper tests, and Python version incompatibility often can’t be detected statically. In the end, we would be relying on someone (possibly an user) reporting that something is broken. Then we’d have to investigate where in the dependency chain the culprit is, and either restrict the new target (and then restrict it in all its reverse dependencies) or immediately fix it.

So in the end, opt-out would be worse for both users and developers. Users would hit package issues first hand, and developers would have to spend significant time on the back-and-forth effort of removing support for new targets, and then adding it again. If we are to add new targets early (which is a worthwhile goal), we have to expect incompatible packages. My experience so far shows that Gentoo developers sometimes end up being the first people to submit patches fixing the incompatibility. This can be a real problem given that many Python packages have slow release cycles, and are blocking their reverse dependencies, and these in turn block their reverse dependencies and so on.

Now, things did not always go smoothly. I have prepared a Python release and Gentoo packaging timeline that puts our packaging work into perspective. As you can see, we were always quite fast in packaging new Python interpreters but it took a significant time to actually switch the default targets to them — in fact, we often switched just before or even after upstream stopped providing bug fixes to the version in question.

Our approach has changed over the years. Early on, we generally kept both the interpreter and the target in ~arch, and stabilized them in order to switch targets. The prolonged stable-mask of the new target has resulted in inconsistent presence of support for the new target in stable, and this in turn involved a lot of last-minute stabilization work. Even then, we ended up switching targets before stable was really ready for that. This was particularly bad for Python 3.4 — the period seen as ‘stable’ on the timeline is actually a period following the first unsuccessful switch of the default. It took us over half a year to try again.

Then (around Python 3.6, if I’m not mistaken) we switched to a different approach. Instead of delaying till the expected switch, we’ve tried to unmask the target on stable systems as soon as possible. This way, we started enforcing dependency graph consistency earlier and were able to avoid big last minute stabilizations needed to unmask the target.

Eventually, thanks to pkgcheck’s superior StableRequestCheck, I’ve started proactively stabilizing new versions of Python packages. This was probably the most important improvement of all. The stabilization effort was streamlined, new versions of packages gained stable keywords sooner and along them did the support for new Python implementations.

The effort in package testing and stabilizations have finally made it possible to catch up with upstream. We have basically gone through a target switching sprint, moving through 3.7 and 3.8 in half a year each. The upcoming switch to Python 3.9 concludes this effort. For the first time in years, our default will be supported upstream for over 6 months.

That said, it is worth noting that things are not becoming easier for us. Over time, Python packages keep getting new dependencies, and this means that every new Python version will involve more and more porting work. Unfortunately, some high profile packages such as setuptools and pytest keep creating bigger and bigger dependency loops. At this point, it is no longer reasonable to attempt to port all the cyclic dependencies simultaneously. Instead, I tend to temporarily disable tests for the initial ports to reduce the number of dependencies. I’ve included a list of suggested initial steps in the Python Guide to ease future porting efforts.

Packaging the Python interpreter

CPython is not the worst thing to package but it’s not the easiest one either. Admittedly, the ebuilds were pretty good back when I joined. However, we’ve already carried a pretty large and mostly undocumented set of patches, and most of these patches we carry up to this day, with no chance of upstreaming them.

Some of the involved patches are build fixes and hacks that are either specific to Gentoo, or bring the flexibility Gentoo cares about (such as making some of the USE flags possible). There are also some fixes for old bugs that upstream has not shown any interest in fixing.

Another part of release management is resolving security bugs. Until recently, we did not track vulnerabilities in CPython very well. Thanks to the work of new Security team members, we have started being informed of vulnerabilities earlier. At the same time, we realized that CPython’s treatment of vulnerabilities is a bit suboptimal.

Admittedly, when very bad things happen upstream releases fixes quickly. However, non-critical vulnerability fixes are released as part of the normal release cycle, approximately every two months. For some time already, every new release contained some security fixes and had to be stabilized quickly. At the same time it contained many other changes with their own breakage potential. Old Python branches were even worse — absurdly, even though these versions received security fixes only, the releases were even rarer.

In the end, I’ve decided that it makes more sense to backport security fixes to our patchset as soon as I become aware of them, and stabilize the patch bumps instead. I do this even if upstream makes a new release at the same time, since patch bumps are safer stable targets. Even then, some of the security fixes actually require changing the behavior. To name a few recent changes:

  • urllib.parse.parse_qsl() historically used to split the URL query string on either & or ;. This somewhat surprising behavior was changed to split only on &, with a parameter to change the separator (but no option to restore the old behavior).
  • urllib.parse.urlparse() historically preserved newlines, CRs and tabs. In the latest Python versions this behavior was changed to follow a newer recommendation of stripping these characters. As a side effect, some URL validators (e.g. in Django) suddenly stopped rejecting URLs with newlines.
  • The ipaddress module recently stopped allowing leading zeros in IPv4 addresses. These were accepted before but some of the external libraries were incidentally interpreting them as octal numbers.

Some of these make you scratch your head.

Unsurprisingly, this is also a lot of work. At this very moment, we are maintaining six different slots of CPython (2.7, 3.6, 3.7, 3.8, 3.9, 3.10). For every security backport set, I start by identifying the security-related commits on the newest release branch. This is easy if they’re accompanied by a news entry in Security category — unfortunately, some vulnerability fixes were treated as regular bug fixes in the past. Once I have a list of commits, I cherry-pick them to our patched branch and make a patchset out of that. This is the easy part.

Now I have to iterate over all the older versions. For maintained branches, the first step is to identify if upstream has already backported their fixes. if they did, I cherry-pick the backport commits from the matching branch. If they did not, I need to verify if the vulnerability applies to this old version, and cherry-pick from a newer version. Sometimes it requires small changes.

The hardest part is Python 2.7 — it is not supported upstream but still used as a build-time dependency by a few projects. The standard library structure in Python 2 differs a lot from the one in Python 3 but many of Python 3 vulnerabilities stem from code dating back to Python 2. In the end, I have to find the relevant piece of original code in Python 2, see if it is vulnerable and usually rewrite the patch for the old code.

I wish this were the end. However, in Gentoo we are also packaging PyPy, the alternative Python implementation. PyPy follows its own release cycle. PyPy2 is based on now-unmaintained Python 2, so most of the vulnerability fixes from our Python 2.7 patchset need to be ported to it. PyPy3 tries to follow Python 3’s standard library but is not very fast at it. In the end, I end up cherry-picking the changes from CPython to PyPy for our patchsets, and at the same time sending them upstream’s way so that they can fix the vulnerabilities without having to sync with CPython immediately.

Historically, we have also supported another alternative implementation, Jython. However, for a very long time upstream’s been stuck on Python 2.5 and we’ve eventually dropped support for Jython as it became unmaintainable. Upstream has eventually caught up with Python 2.7 but you can imagine how useful that is today.

There are other interesting projects out there but I don’t think we have the manpower (or even a very good reason) to work on them.

Now, a fun fact: Victor Stinner indicates in his mail Need help to fix known Python security vulnerabilities that the security fixes CPython receives are only a drop in the ocean. Apparently, at the time of writing there were 78 open problems of various severity.

A bus and a phoenix

Any project where a single developer does a sufficiently large portion of work has a tendency towards a bus factor of one. The Python project was no different. Ten years ago we were relying on a single person doing most of the work. When that person retired, things started going out of hand. Python ended up with complex eclasses that nobody wholly understood and that urgently needed updates. Many packages were getting outdated.

I believe that we did the best that we could at the time. We’ve started the new eclass set because it was much easier to draw a clear line and start over. We ended up removing many of the less important or more problematic packages, from almost the whole net-zope category (around 200 packages) to temporarily removing Django (only to readd it later on). Getting everything at least initially ported took a lot of time. Many of the ports turned out buggy, a large number of packages were outdated, unmaintained, missing test suites.

Today I can finally say that the Python team’s packages standing is reasonably good. However, in order for it to remain good we need to put a lot of effort every day. Packages need to be bumped. Bumps often add new dependencies. New Python versions require testing. Test regressions just keep popping up, and often when you’re working on something else. Stabilizations need to be regularly tracked, and they usually uncover even more obscure test problems. Right now it’s pretty normal for me to spend an hour of my time every day just to take care of the version bumps.

I have tried my best to avoid returning to a bus factor of one. I have tried to keep the eclasses simple (but I can’t call that a complete success), involve more people in the development, keep things well documented and packages clean. Yet I still end up doing a very large portion of work in the team. I know that there are other developers who can stand in for me but I’m not sure if they will be able to take up all the work, given all their other duties.

We really need young blood. We need people who would be able to dedicate a lot of time specifically to Python, and learn all the intricacies of Python packaging. While I’ve done my best to document everything I can think of in the Python Guide, it’s still very little. The Python ecosystem is a very diverse one, and surprisingly hard to maintain without burning out. Of course, there are many packages that are pleasure to maintain, and many upstreams that are pleasure to work with. Unfortunately, there are also many projects that make you really frustrated — with bad quality code, broken tests, lots of NIH dependencies and awful upstreams that simply hate packagers.

Over my 10 (and a half) years as a developer, I have done a lot of different things. However, if I were to point one area of Gentoo where I put most of my effort, that area would be Python. I am proud of all that we’ve been able to accomplish, and how great our team is. However, there are still many challenges ahead of us, as well as a lot of tedious work.

I’m a Gentoo developer for over 10 years already. I’ve been doing a lot of different things throughout that period. However, Python was pretty much always somewhere within my area of interest. I don’t really recall how it all started. Maybe it had something to do with Portage being written in Python. Maybe it was the natural next step after programming in Perl.

I feel like the upcoming switch to Python 3.9 is the last step in the prolonged effort of catching up with Python. Over the last years, we’ve been working real hard to move Python support forward, to bump neglected packages, to enable testing where tests are available, to test packages on new targets and unmask new targets as soon as possible. We have improved the processes a lot. Back when we were switching to Python 3.4, it took almost a year from the first false start attempt to the actual change. We started using Python 3.5 by default after upstream dropped bugfix support for it. In a month from now, we are going to start using Python 3.9 even before 3.10 final is released.

I think this is a great opportunity to look back and see what changed in the Gentoo Python ecosystem, in the last 10 years.

Python package ebuilds 10 years ago

Do you know how a Python package ebuild looked like 10 years ago? Let’s take gentoopm-0.1 as an example (reformatted to fit the narrow layout better):


# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.1.ebuild,v 1.1 2011/07/15 19:05:26 mgorny Exp $

EAPI=3

PYTHON_DEPEND='*:2.6'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS='2.4 2.5'
DISTUTILS_SRC_TEST=setup.py

inherit base distutils

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/gentoopm/"
SRC_URI="http://cloud.github.com/downloads/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.8.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="dev-python/epydoc"
PDEPEND="app-admin/eselect-package-manager"

src_prepare() {
  base_src_prepare
  distutils_src_prepare
}

src_compile() {
  distutils_src_compile

  if use doc; then
    "$(PYTHON -2)" setup.py doc || die
  fi
}

src_install() {
  distutils_src_install

  if use doc; then
    dohtml -r doc/* || die
  fi
}

This ebuild is actually using the newer API of python.eclass that is enabled via SUPPORT_PYTHON_ABIS. It provides support for installing for multiple implementations (like the modern python-r1 eclass). PYTHON_DEPEND is used to control the dependency string added to ebuild. The magical syntax here means that the ebuild supports both Python 2 and Python 3, from Python 2.6 upwards. RESTRICT_PYTHON_ABIS opts out support for Python versions prior to 2.6. Note the redundancy — PYTHON_DEPEND controls the dependency, specified as a range of Python 2 and/or Python 3 versions, RESTRICT_PYTHON_ABIS controls versions used at build time and needs to explicitly exclude all unsupported branches.

Back then, there were no PYTHON_TARGETS to control what was built. Instead, the eclass defaulted to using whatever was selected via eselect python, with the option to override it via setting USE_PYTHON in make.conf. Therefore, there were no cross-package USE dependencies and you had to run python-updater to verify whether all packages are built for the current interpreter, and rebuild these that were not.

Still, support for multiple ABIs, as the eclass called different branches/implementations of Python, was a major step forward. It was added around the time that the first releases of Python 3 were published, and our users have been relying on it to support a combination of Python 2 and Python 3 for years. Today, we’re primarily using it to aid developers in testing their packages and to provide a safer upgrade experience.

The python.eclass stalemate

Unfortunately, things at the time were not all that great. Late 2010 marks a conflict between the primary Python developer and the rest of the community, primarily due to the major breakage being caused by the changes in Python support. By mid-2011, it was pretty clear that there is no chance to resolve the conflict. The in-Gentoo version of python.eclass was failing to get EAPI 4 support for 6 months already, while an incompatible version continued being developed in the (old) Python overlay. As Dirkjan Ochtman related in his mail:

I guess by now pretty much everyone knows that the python eclass is rather complex, and that this poses some problems. This has also been an important cause for the disagreements between Arfrever and some of the other developers. Since it appears that Arfrever won’t be committing much code to gentoo-x86 in the near future, I’m trying to figure out where we should go with the python.eclass. […]

Dirkjan Ochtman, 2011-06-27, [gentoo-dev] The Python problem

Eventually, some of the changes from the Python overlay were backported to the eclass and EAPI 4 support was added. Nevertheless, at this point it was pretty clear that we need a new way forward. Unfortunately, the discussions were leading nowhere. With the primary eclass maintainer retired, nobody really comprehended most of the eclass, nor were able to afford the time to figure it out. At the same time, involved parties wanted to preserve backwards compatibility while moving forward.

The tie breaker: python-distutils-ng

Some of you might find it surprising that PYTHON_TARGETS are not really a python-r1 invention. Back in March 2012, when Python team was still unable to find a way forward with python.eclass, Krzysztof Pawlik (nelchael) has committed a new python-distutils-ng.eclass. It has never grown popular, and it has been replaced by the python-r1 suite before it ever started being a meaningful replacement for python.eclass. Still, it served an important impulse that made what came after possible.

Here’s a newer gentoopm ebuild using the new eclass (again reformatted):


# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.2.5-r1.ebuild,v 1.1 2012/05/26 10:11:21 mgorny Exp $

EAPI=4
PYTHON_COMPAT='python2_6 python2_7 python3_1 python3_2'

inherit base python-distutils-ng

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/mgorny/gentoopm/"
SRC_URI="mirror://github/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~mips ~x86 ~x86-fbsd"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.10.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="doc? ( dev-python/epydoc )"
PDEPEND="app-admin/eselect-package-manager"

python_prepare_all() {
  base_src_prepare
}

src_compile() {
  python-distutils-ng_src_compile
  if use doc; then
    "${PYTHON}" setup.py doc || die
  fi
}

python_install_all() {
  if use doc; then
    dohtml -r doc/*
  fi
}

Just looking at the code, you may see that python-r1 has inherited a lot after this eclass. python-distutils-ng in turn followed some of the good practices introduced before in the ruby-ng eclass. It introduced PYTHON_TARGETS to provide explicit visible control over implementations used for the build — though notably it did not include a way for packages to depend on matching flags (i.e. the equivalent of ${PYTHON_USEDEP}). It also used the sub-phase approach that makes distutils-r1 and ruby-ng eclasses much more convenient than the traditional python.eclass approach that roughly resembled using python_foreach_impl all the time.

What’s really important is that python-distutils-ng carved a way forward. It’s been a great inspiration and a proof of concept. It has shown that we do not have to preserve compatibility with python.eclass forever, or have to learn its inner workings before starting to solve problems. I can’t say how Python would look today if it did not happen but I can say with certainly that python-r1 would not happen so soon if it weren’t for it.

python-r1

In October 2012, the first version of python-r1 was committed. It combined some of the very good ideas of python-distutils-ng with some of my own.  The goal was not to provide an immediate replacement for python.eclass. Instead, the plan was to start simple and add new features as they turned out to be necessary. Not everything went perfectly but I dare say that the design has stood the test of time. While I feel like the eclasses ended up being more complex than I wished they would be, they still work fine with no replacement in sight and they serve as inspiration to other eclasses.

For completeness, here’s a 2017 gentoopm live ebuild that uses a pretty complete distutils-r1 feature set:


# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6
PYTHON_COMPAT=( python{2_7,3_4,3_5,3_6} pypy )

EGIT_REPO_URI="https://github.com/mgorny/gentoopm.git"
inherit distutils-r1 git-r3

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/mgorny/gentoopm/"
SRC_URI=""

LICENSE="BSD-2"
SLOT="0"
KEYWORDS=""
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/pkgcore-0.9.4[${PYTHON_USEDEP}]
    >=sys-apps/portage-2.1.10.3[${PYTHON_USEDEP}]
    >=sys-apps/paludis-3.0.0_pre20170219[python,${PYTHON_USEDEP}]
  )"
DEPEND="
  doc? (
    dev-python/epydoc[$(python_gen_usedep python2_7)]
  )"
PDEPEND="app-eselect/eselect-package-manager"

REQUIRED_USE="
  doc? ( $(python_gen_useflags python2_7) )"

src_configure() {
  use doc && DISTUTILS_ALL_SUBPHASE_IMPLS=( python2.7 )
  distutils-r1_src_configure
}

python_compile_all() {
  use doc && esetup.py doc
}

python_test() {
  esetup.py test
}

python_install_all() {
  use doc && local HTML_DOCS=( doc/. )
  distutils-r1_python_install_all
}

The new eclasses have employed the same PYTHON_TARGETS flags for the general implementation choice but also added PYTHON_SINGLE_TARGET to make choosing the implementation more convenient (and predictable at the same time) when the package did not permit choosing more than one. The distutils-r1 eclass reused the great idea of sub-phases to make partial alterations to the phases easier.

The unique ideas included:

  • the split into more eclasses by functionality (vs switching the mode via variables as done in python.eclass)
  • exposing dependencies and REQUIRED_USE constraints via variables to be used in ebuild instead of writing elaborate mechanisms for adding dependencies
  • avoiding command substitution in global scope to keep metadata regeneration fast

The eclasses evolved a lot over the years. Some of the original ideas turned out pretty bad, e.g. trying to run sub-phases in parallel (which broke a lot of stuff for minor performance gain) or the horribly complex original interaction between PYTHON_TARGETS and PYTHON_SINGLE_TARGET. Ebuilds often missed dependencies and REQUIRED_USE constraints, until we finally made the pkgcheck-based CI report that.

The migration to new eclasses took many years. Initially, python-r1 ebuilds were even depending on python.eclass ebuilds. The old eclass was not removed until March 2017, i.e. 4.5 years after introducing the new eclasses.

Testing on new Python targets

We went for the opt-in approach with PYTHON_COMPAT. This means that for every new Python target added, we start with no packages supporting it and have to iterate over all of the packages adding the support. It’s a lot of work and it has repeatedly caused users pain due to packages not being ported in time for the big switch to the next version. Some people have complained about that and suggested that we should go for opt-out instead. However, if you think about it, opt-in is the only way to go.

The big deal is that for any particular package to support a new implementation, all of its Python dependencies need to support it as well. With the opt-in approach, it means that we’re doing the testing dependency-first, and reaching the bigger packages only when we confirm that there’s at least a single version of every dependency that works for it. If we do things right, users don’t even see any regressions.

If we went for the opt-out approach, all packages would suddenly claim to support the new version. Now, this wouldn’t be that bad if we were actually able to do a big CI run for all packages — but we can’t since a lot of them do not have proper tests, and Python version incompatibility often can’t be detected statically. In the end, we would be relying on someone (possibly an user) reporting that something is broken. Then we’d have to investigate where in the dependency chain the culprit is, and either restrict the new target (and then restrict it in all its reverse dependencies) or immediately fix it.

So in the end, opt-out would be worse for both users and developers. Users would hit package issues first hand, and developers would have to spend significant time on the back-and-forth effort of removing support for new targets, and then adding it again. If we are to add new targets early (which is a worthwhile goal), we have to expect incompatible packages. My experience so far shows that Gentoo developers sometimes end up being the first people to submit patches fixing the incompatibility. This can be a real problem given that many Python packages have slow release cycles, and are blocking their reverse dependencies, and these in turn block their reverse dependencies and so on.

Now, things did not always go smoothly. I have prepared a Python release and Gentoo packaging timeline that puts our packaging work into perspective. As you can see, we were always quite fast in packaging new Python interpreters but it took a significant time to actually switch the default targets to them — in fact, we often switched just before or even after upstream stopped providing bug fixes to the version in question.

Our approach has changed over the years. Early on, we generally kept both the interpreter and the target in ~arch, and stabilized them in order to switch targets. The prolonged stable-mask of the new target has resulted in inconsistent presence of support for the new target in stable, and this in turn involved a lot of last-minute stabilization work. Even then, we ended up switching targets before stable was really ready for that. This was particularly bad for Python 3.4 — the period seen as ‘stable’ on the timeline is actually a period following the first unsuccessful switch of the default. It took us over half a year to try again.

Then (around Python 3.6, if I’m not mistaken) we switched to a different approach. Instead of delaying till the expected switch, we’ve tried to unmask the target on stable systems as soon as possible. This way, we started enforcing dependency graph consistency earlier and were able to avoid big last minute stabilizations needed to unmask the target.

Eventually, thanks to pkgcheck’s superior StableRequestCheck, I’ve started proactively stabilizing new versions of Python packages. This was probably the most important improvement of all. The stabilization effort was streamlined, new versions of packages gained stable keywords sooner and along them did the support for new Python implementations.

The effort in package testing and stabilizations have finally made it possible to catch up with upstream. We have basically gone through a target switching sprint, moving through 3.7 and 3.8 in half a year each. The upcoming switch to Python 3.9 concludes this effort. For the first time in years, our default will be supported upstream for over 6 months.

That said, it is worth noting that things are not becoming easier for us. Over time, Python packages keep getting new dependencies, and this means that every new Python version will involve more and more porting work. Unfortunately, some high profile packages such as setuptools and pytest keep creating bigger and bigger dependency loops. At this point, it is no longer reasonable to attempt to port all the cyclic dependencies simultaneously. Instead, I tend to temporarily disable tests for the initial ports to reduce the number of dependencies. I’ve included a list of suggested initial steps in the Python Guide to ease future porting efforts.

Packaging the Python interpreter

CPython is not the worst thing to package but it’s not the easiest one either. Admittedly, the ebuilds were pretty good back when I joined. However, we’ve already carried a pretty large and mostly undocumented set of patches, and most of these patches we carry up to this day, with no chance of upstreaming them.

Some of the involved patches are build fixes and hacks that are either specific to Gentoo, or bring the flexibility Gentoo cares about (such as making some of the USE flags possible). There are also some fixes for old bugs that upstream has not shown any interest in fixing.

Another part of release management is resolving security bugs. Until recently, we did not track vulnerabilities in CPython very well. Thanks to the work of new Security team members, we have started being informed of vulnerabilities earlier. At the same time, we realized that CPython’s treatment of vulnerabilities is a bit suboptimal.

Admittedly, when very bad things happen upstream releases fixes quickly. However, non-critical vulnerability fixes are released as part of the normal release cycle, approximately every two months. For some time already, every new release contained some security fixes and had to be stabilized quickly. At the same time it contained many other changes with their own breakage potential. Old Python branches were even worse — absurdly, even though these versions received security fixes only, the releases were even rarer.

In the end, I’ve decided that it makes more sense to backport security fixes to our patchset as soon as I become aware of them, and stabilize the patch bumps instead. I do this even if upstream makes a new release at the same time, since patch bumps are safer stable targets. Even then, some of the security fixes actually require changing the behavior. To name a few recent changes:

  • urllib.parse.parse_qsl() historically used to split the URL query string on either & or ;. This somewhat surprising behavior was changed to split only on &, with a parameter to change the separator (but no option to restore the old behavior).
  • urllib.parse.urlparse() historically preserved newlines, CRs and tabs. In the latest Python versions this behavior was changed to follow a newer recommendation of stripping these characters. As a side effect, some URL validators (e.g. in Django) suddenly stopped rejecting URLs with newlines.
  • The ipaddress module recently stopped allowing leading zeros in IPv4 addresses. These were accepted before but some of the external libraries were incidentally interpreting them as octal numbers.

Some of these make you scratch your head.

Unsurprisingly, this is also a lot of work. At this very moment, we are maintaining six different slots of CPython (2.7, 3.6, 3.7, 3.8, 3.9, 3.10). For every security backport set, I start by identifying the security-related commits on the newest release branch. This is easy if they’re accompanied by a news entry in Security category — unfortunately, some vulnerability fixes were treated as regular bug fixes in the past. Once I have a list of commits, I cherry-pick them to our patched branch and make a patchset out of that. This is the easy part.

Now I have to iterate over all the older versions. For maintained branches, the first step is to identify if upstream has already backported their fixes. if they did, I cherry-pick the backport commits from the matching branch. If they did not, I need to verify if the vulnerability applies to this old version, and cherry-pick from a newer version. Sometimes it requires small changes.

The hardest part is Python 2.7 — it is not supported upstream but still used as a build-time dependency by a few projects. The standard library structure in Python 2 differs a lot from the one in Python 3 but many of Python 3 vulnerabilities stem from code dating back to Python 2. In the end, I have to find the relevant piece of original code in Python 2, see if it is vulnerable and usually rewrite the patch for the old code.

I wish this were the end. However, in Gentoo we are also packaging PyPy, the alternative Python implementation. PyPy follows its own release cycle. PyPy2 is based on now-unmaintained Python 2, so most of the vulnerability fixes from our Python 2.7 patchset need to be ported to it. PyPy3 tries to follow Python 3’s standard library but is not very fast at it. In the end, I end up cherry-picking the changes from CPython to PyPy for our patchsets, and at the same time sending them upstream’s way so that they can fix the vulnerabilities without having to sync with CPython immediately.

Historically, we have also supported another alternative implementation, Jython. However, for a very long time upstream’s been stuck on Python 2.5 and we’ve eventually dropped support for Jython as it became unmaintainable. Upstream has eventually caught up with Python 2.7 but you can imagine how useful that is today.

There are other interesting projects out there but I don’t think we have the manpower (or even a very good reason) to work on them.

Now, a fun fact: Victor Stinner indicates in his mail Need help to fix known Python security vulnerabilities that the security fixes CPython receives are only a drop in the ocean. Apparently, at the time of writing there were 78 open problems of various severity.

A bus and a phoenix

Any project where a single developer does a sufficiently large portion of work has a tendency towards a bus factor of one. The Python project was no different. Ten years ago we were relying on a single person doing most of the work. When that person retired, things started going out of hand. Python ended up with complex eclasses that nobody wholly understood and that urgently needed updates. Many packages were getting outdated.

I believe that we did the best that we could at the time. We’ve started the new eclass set because it was much easier to draw a clear line and start over. We ended up removing many of the less important or more problematic packages, from almost the whole net-zope category (around 200 packages) to temporarily removing Django (only to readd it later on). Getting everything at least initially ported took a lot of time. Many of the ports turned out buggy, a large number of packages were outdated, unmaintained, missing test suites.

Today I can finally say that the Python team’s packages standing is reasonably good. However, in order for it to remain good we need to put a lot of effort every day. Packages need to be bumped. Bumps often add new dependencies. New Python versions require testing. Test regressions just keep popping up, and often when you’re working on something else. Stabilizations need to be regularly tracked, and they usually uncover even more obscure test problems. Right now it’s pretty normal for me to spend an hour of my time every day just to take care of the version bumps.

I have tried my best to avoid returning to a bus factor of one. I have tried to keep the eclasses simple (but I can’t call that a complete success), involve more people in the development, keep things well documented and packages clean. Yet I still end up doing a very large portion of work in the team. I know that there are other developers who can stand in for me but I’m not sure if they will be able to take up all the work, given all their other duties.

We really need young blood. We need people who would be able to dedicate a lot of time specifically to Python, and learn all the intricacies of Python packaging. While I’ve done my best to document everything I can think of in the Python Guide, it’s still very little. The Python ecosystem is a very diverse one, and surprisingly hard to maintain without burning out. Of course, there are many packages that are pleasure to maintain, and many upstreams that are pleasure to work with. Unfortunately, there are also many projects that make you really frustrated — with bad quality code, broken tests, lots of NIH dependencies and awful upstreams that simply hate packagers.

Over my 10 (and a half) years as a developer, I have done a lot of different things. However, if I were to point one area of Gentoo where I put most of my effort, that area would be Python. I am proud of all that we’ve been able to accomplish, and how great our team is. However, there are still many challenges ahead of us, as well as a lot of tedious work.

March 26 2021

LetsEncrypt SSL certificates for vhosts within the mail stack – via Postfix (SMTP) and Dovecot (IMAP)

Nathan Zachary (nathanzachary) March 26, 2021, 4:36

For a long time, I didn’t care about using self-signed SSL certificates for the mail stack because 1) they still secured the connection to the server, and 2) those certificates weren’t seen or utilised by anyone other than me. However, for my datacentre infrastructure which houses clients’ websites and email, using self-signed or generic certificates (even for the mail stack) wasn’t a very good solution as mail clients (e.g Thunderbird or Outlook) notify users of the problematic certs. Clients with dedicated mail servers could use valid certificates (freely from LetsEncrypt) without problem, but those on shared infrastructure posed a different issue—how can mail for different domains all sharing the same IPv4 address use individualised certificates for their SMTP and IMAP connections? This article will explain the method that I used to assign domain-specific certificates for the full mail stack using LetsEncrypt’s certbot for the certs themselves, the Postfix MTA (mail transfer agent [for SMTP]), and the Dovecot IMAP server. This article is tailored to Gentoo Linux, but should be easily applied to nearly any distribution. Two general caveats are:

  • The locations of files may be different for your distribution. Consult your distribution’s documentation for the appropriate file locations
  • I use OpenRC as my init system in Gentoo. If your distribution uses systemd, you will need to replace any reference to code snippets containing /etc/init.d/$SERVICE $ACTION with systemctl $ACTION $SERVICE
    • As an example, I mention restarting Dovecot with /etc/init.d/dovecot restart
    • A systemd user would instead issue systemctl restart dovecot

I will provide a significant amount of ancillary information pertaining to each step of this process. If you are comfortable with the mail stack, some of it may be rudimentary, so feel free to skip ahead. Conversely, if any of the concepts are new or foreign to you, please reference Gentoo’s documentation for setting up a Complete Virtual Mail Server as a prerequisite. For the remainder of the article, I am going to use two hypothetical domains of domain1.com and domain2.com. Any time that they are referenced, you will want to replace them with your actual domains.

BIND (DNS) configurations

In order to keep the web stack and mail stack separate in terms of DNS, I like to have mail.domain1.com & mail.domain2.com subdomains for the MX records, but just have them point to the same A record used for the website. Some may consider this to be unnecessary, but I have found the separation helpful for troubleshooting if any problems should arise. Here are the relevant portions of the zone files for each domain:

# grep -e 'A\|MX ' /etc/bind/pri/domain1.com.external.zone 
domain1.com.		300	IN	A	$IP_ADDRESS
mail.domain1.com.	300	IN	A	$IP_ADDRESS

# grep -e 'A\|MX ' /etc/bind/pri/domain2.com.external.zone 
domain2.com.		300	IN	A	$IP_ADDRESS
mail.domain2.com.	300	IN	A	$IP_ADDRESS

In the above snippets, $IP_ADDRESS should be the actual IPv4 address of the webserver. It should be noted that, in this setup, the web stack and the mail stack reside on the same physical host, so the IP is the same for both stacks.

Apache (webserver) configurations

As mentioned above, I keep the web stack and mail stack separate in terms of DNS. For the LetsEncrypt certificates (covered in the next section), though, I use the same certificate for both stacks. I do so by generating the cert for both the main domain and the ‘mail’ subdomain. In order for this to work, I make the ‘mail’ subdomain a ServerAlias in the Apache vhost configurations:

# grep -e 'ServerName \|ServerAlias ' www.domain1.com.conf 
	ServerName domain1.com
	ServerAlias www.domain1.com mail.domain1.com

# grep -e 'ServerName \|ServerAlias ' www.domain2.com.conf 
	ServerName domain2.com
	ServerAlias www.domain2.com mail.domain2.com

This allows the verification of the ‘mail’ subdomain to be done via the main URL instead of requiring a separate public-facing site directory for it.

LetsEncrypt SSL certificates

LetsEncrypt is a non-profit certificate authority (CA) that provides X.509 (TLS) certificates free-of-charge. The issued certificates are only valid for 90 days, which encourages automated processes to handle renewals. The recommended method is to use the certbot tool for renewals, and there are many plugins available that provide integration with various webservers. Though I run a combination of Apache and NGINX, I prefer to not have certbot directly interact with them. Rather, I choose to rely on certbot solely for the certificate generation & renewal, and to handle the installation thereof via other means. For this tutorial, I will use the ‘certonly’ option with the webroot plugin:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/$DOMAIN/$HOST/htdocs/ --domains $DOMAIN,$DOMAIN,$DOMAIN

In the code snippet above, you will replace $DOMAIN with the actual domain, and $HOST with the subdomain. So, for our two hypothetical domains, the commands translate as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,mail.domain1.com

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain2.com/www/htdocs/ --domains domain2.com,www.domain2.com,mail.domain2.com

The webroot plugin will create a temporary file under ${webroot-path}/.well-known/acme-challenge/ and then check that file via HTTP in order to validate the server. Make sure that the directory is publicly accessible or else the validation will fail. Once certbot validates the listed domains—in this setup, the ‘www’ and ‘mail’ subdomains are just aliases to the primary domain (see the BIND and Apache configurations sections above)—it will generate the SSL certificates and place them under /etc/letsencrypt/live/domain1.com/ and /etc/letsencrypt/live/domain2.com/, respectively. There will be two files for each certificate:

  • fullchain.pem –> the public certificate
  • privkey.pem –> the private key for the certificate

Though these certificates are often used in conjunction with the web stack, we are going to use them for securing the mail stack as well.

Dovecot (IMAP) configurations

Now that we have the certificates for each domain, we’ll start by securing the IMAP server (i.e. Dovecot) so that the users’ Mail User Agent (MUA, or more colloquially, “email client” [like Thunderbird or Outlook]) will no longer require a security exception due to a domain mismatch. Adding the domain-specific SSL certificate to Dovecot is a straightforward process that only requires two directives per domain. For domain1.com and domain2.com, add the following lines to /etc/dovecot/conf.d/10-ssl.conf:

local_name mail.domain1.com {
  ssl_cert = </etc/letsencrypt/live/domain1.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain1.com/privkey.pem
}

local_name mail.domain2.com {
  ssl_cert = </etc/letsencrypt/live/domain2.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain2.com/privkey.pem
}

Those code blocks can be copied and pasted for any additional virtual hosts or domains that are needed. As with any configuration change, make sure to restart the application in order to make the changes active:

/etc/init.d/dovecot restart
Postfix (SMTP) configurations

Configuring Dovecot to use the SSL certificate for securing the IMAP connection from the user’s email client to the server is only one part of the process—namely the connection when the user is retrieving mails from the server. This next part will use the same certificate to secure the SMTP connection (via the Postfix SMTP server) for sending mails.

The first step is to create a file that will be used for mapping each certificate to its respective domain. Postfix can handle this correlation via Server Name Indication (SNI), which is an extension of the TLS protocol that indicates the hostname of the server at the beginning of the handshake process. Though there is no naming requirement for this map file, I chose to create it as /etc/postfix/vmail_ssl. The format of the file is:

$DOMAIN   $PRIVATE_KEY   $PUBLIC_KEY_CHAIN

So for our example of domain1.com and domain2.com, the file would consist of the following entries:

mail.domain1.com /etc/letsencrypt/live/domain1.com/privkey.pem /etc/letsencrypt/live/domain1.com/fullchain.pem

mail.domain2.com /etc/letsencrypt/live/domain2.com/privkey.pem /etc/letsencrypt/live/domain2.com/fullchain.pem

Though this file is in plaintext, Postfix doesn’t understand the mapping in this format. Instead, a base64-encoded Berkeley DB file needs to be used. Thankfully, Postfix makes creating such a file very easy via the postmap utility. Once you have created and populated your /etc/postfix/vmail_ssl file with the entries for each domain, issue the following command:

# postmap -F hash:/etc/postfix/vmail_ssl

which will create the Berkeley DB file (named vmail_ssl.db) in the same directory:

# find /etc/postfix/ -type f -iname '*vmail_ssl*'
/etc/postfix/vmail_ssl.db
/etc/postfix/vmail_ssl

# file /etc/postfix/vmail_ssl
/etc/postfix/vmail_ssl: ASCII text

# file /etc/postfix/vmail_ssl.db 
/etc/postfix/vmail_ssl.db: Berkeley DB (Hash, version 10, native byte-order)

Now that we have created the mapping table, we have to configure Postfix to use it for SMTP connections. It’s acceptable to have both the SNI-mapped certificates AND a generic SSL certificate as the default (for when a domain isn’t listed in the mapping table). Postfix can have both directives specified simultaneously. To do so, the following directives need to be added to /etc/postfix/main.cf (the comments explain both sets of directives):

## Default SSL cert for SMTP if SNI is not enabled
smtpd_tls_cert_file = /etc/ssl/mail/server.pem
smtpd_tls_key_file = /etc/ssl/mail/server.key

## Mappings for SMTP SSL certs when SNI is enabled
tls_server_sni_maps = hash:/etc/postfix/vmail_ssl

After making those modifications to Postfix’s main configuration file, it is required to restart it:

/etc/init.d/postfix restart

That’s it! Now the full mail stack is secured using domain-specific SSL certificates for both IMAP and SMTP connections. The remaining sections below will explain some maintenance-related procedures such as handling the LetsEncrypt certificate renewals & updating the mappings in Postfix automatically, as well as verifying it’s all working as intended (and some troubleshooting tips in case it’s not). ♦

Automatic renewal (cron) configurations

As mentioned in the LetsEncrypt section above, the certificates that they issue are only valid for a period of 90 days. One of the reasons for the relatively short validity period is to encourage automation when it comes to renewing them. I choose to handle the renewals automatically via cron:

# tail -n 4 /var/spool/cron/crontabs/root 

## LetsEncrypt certificate renewals on first of each month
## See /etc/letsencrypt/renewal-hooks/post/ for Postfix & Apache hooks 
0  2  1  *  *   /usr/bin/certbot renew --quiet

This cron entry instructs LetsEncrypt’s certbot to check the validity of ALL certificates at 02:00 (server time) on the first of every month (if that format is unfamiliar to you, see Wikipedia’s article on cron). The renew subcommand will automatically generate a new certificate for any found to expire within the next 30 days, and the quiet option will silence any output except for errors, which is appropriate for use with a cron job.

That’s the procedure for renewing the certificate automatically, but what about then automatically updating the appropriate stack configurations—in particular, Postfix’s vmail_ssl mappings table (and Apache, but that’s outside the scope of this tutorial)? If the certificate is renewed, but it is not updated in Postfix’s hash table, there will be a mismatch error. As mentioned in the comment on the cron entry, I chose to handle those configuration updates automatically via certbot’s ‘renewal hooks’, which can be found under /etc/letsencrypt/renewal-hooks/. In this case, the configuration updates need to happen after certificate renewal, so they are put under the post/ subdirectory.

I have two scripts that run after a certificate renewal, but only the 01_postfix_smtp_ssl.sh one is applicable for the mail stack:

# ls /etc/letsencrypt/renewal-hooks/post/
01_postfix_smtp_ssl.sh  02_apache.sh

# cat /etc/letsencrypt/renewal-hooks/post/01_postfix_smtp_ssl.sh 
#!/bin/bash
/usr/sbin/postmap -F hash:/etc/postfix/vmail_ssl
/etc/init.d/postfix restart
exit 0

The simple script issues the same postmap command from the ‘Postfix (SMTP) configurations‘ section above, and then restarts Postfix. If everything goes smoothly, it will exit cleanly (‘exit 0’). The script ensures that the new certificate is immediately applied to the Postfix configuration so that there aren’t validation errors after the automated renewal process.

Verification of the certificates

If everything went according to plan, valid SSL certificates should be in place for both mail.domain1.com and mail.domain2.com. Like any good engineer, though, we don’t want to just assume that it’s working as intended. So… we should test it! You could just open an email client of your choice and view the certificates for IMAP and SMTP connections. Personally, though, I prefer using terminal-based utilities as I find them to be more efficient. In this case, we can use the openssl command for connecting to each domain as a test, and the basic syntax is:

For SMTP:

openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp

For IMAP:

openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com

These commands will output a lot of information including the full public certificate, the issuing authority (LetsEncrypt), handshake details, SSL session details, and so on. If you’re interested in all of those details, feel free to issue the commands as they are above (obviously swapping out the actual domains and the ports that you use for SMTP and IMAP). If, however, you simply want to confirm that the certificates are valid, you can pipe the commands to grep in order to limit the output:

For SMTP:

$ openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
250 CHUNKING
subject=CN = domain1.com
Verify return code: 0 (ok)

For IMAP:

$ openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
subject=CN = domain1.com
Verify return code: 0 (ok)

If you see output similar to what’s above, then everything is working as it should be. In particular, you want to make sure that references to the ‘CN’ match the domain, and that you see a ‘Verify return code:’ of 0 (ok) Pat yourself on the back and grab your beverage of choice to celebrate a job well done. ♦

Additional information

If you have already been using a domain for a website or other service, chances are that you have already generated a LetsEncrypt SSL certificate for it. Thankfully LetsEncrypt makes it easy to append a new subdomain to an existing certificate instead of having to generate a completely separate one for the ‘mail’ subdomain used in this guide (e.g. mail.domain1.com).

The first step is to find the certificate that you want to modify (in this case, domain1.com) and see which subdomains are covered under it. This can be accomplished using the certbot certificates command. The output will look something like this:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com
   Expiry Date: 2021-05-02 06:03:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

The important part is the list of subdomains on the Domains: line because you need to reference ALL of them when using the --expand flag that follows. Using the output from above, the command would be constructed as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,staging.domain1.com,mail.domain1.com --expand

If certbot indicates that the new certificate has been generated without any errors, you can check it again using the certbot certificates command from above and validate that now the ‘mail’ subdomain is listed as well:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com mail.domain1.com
   Expiry Date: 2021-05-14 06:05:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem
Troubleshooting

I can’t anticipate the full gamut of problems that could potentially arise when going through this guide, but I will try to cover some common pitfalls here. If you run into a problem, feel free to comment and I will try to help you through it.

>>> Postfix error about the table hash:

If Postfix won’t start after the modifications from the sections above, and you see a line like this in the mail logs:

[postfix/smtpd] warning: table hash:/etc/postfix/vmail_ssl.db: key mail.domain1.com: malformed BASE64 value: /etc/letsencrypt/live/domain1

then the problem stems from running postmap without the -F flag. Try it again with that flag: postmap -F hash:/etc/postfix/vmail_ssl which should create a syntactically correct hash table, allowing Postfix to properly start up.

For a long time, I didn’t care about using self-signed SSL certificates for the mail stack because 1) they still secured the connection to the server, and 2) those certificates weren’t seen or utilised by anyone other than me. However, for my datacentre infrastructure which houses clients’ websites and email, using self-signed or generic certificates (even for the mail stack) wasn’t a very good solution as mail clients (e.g Thunderbird or Outlook) notify users of the problematic certs. Clients with dedicated mail servers could use valid certificates (freely from LetsEncrypt) without problem, but those on shared infrastructure posed a different issue—how can mail for different domains all sharing the same IPv4 address use individualised certificates for their SMTP and IMAP connections? This article will explain the method that I used to assign domain-specific certificates for the full mail stack using LetsEncrypt’s certbot for the certs themselves, the Postfix MTA (mail transfer agent [for SMTP]), and the Dovecot IMAP server. This article is tailored to Gentoo Linux, but should be easily applied to nearly any distribution. Two general caveats are:

  • The locations of files may be different for your distribution. Consult your distribution’s documentation for the appropriate file locations
  • I use OpenRC as my init system in Gentoo. If your distribution uses systemd, you will need to replace any reference to code snippets containing /etc/init.d/$SERVICE $ACTION with systemctl $ACTION $SERVICE
    • As an example, I mention restarting Dovecot with /etc/init.d/dovecot restart
    • A systemd user would instead issue systemctl restart dovecot

I will provide a significant amount of ancillary information pertaining to each step of this process. If you are comfortable with the mail stack, some of it may be rudimentary, so feel free to skip ahead. Conversely, if any of the concepts are new or foreign to you, please reference Gentoo’s documentation for setting up a Complete Virtual Mail Server as a prerequisite. For the remainder of the article, I am going to use two hypothetical domains of domain1.com and domain2.com. Any time that they are referenced, you will want to replace them with your actual domains.

BIND (DNS) configurations

In order to keep the web stack and mail stack separate in terms of DNS, I like to have mail.domain1.com & mail.domain2.com subdomains for the MX records, but just have them point to the same A record used for the website. Some may consider this to be unnecessary, but I have found the separation helpful for troubleshooting if any problems should arise. Here are the relevant portions of the zone files for each domain:

# grep -e 'A\|MX ' /etc/bind/pri/domain1.com.external.zone 
domain1.com.		300	IN	A	$IP_ADDRESS
mail.domain1.com.	300	IN	A	$IP_ADDRESS

# grep -e 'A\|MX ' /etc/bind/pri/domain2.com.external.zone 
domain2.com.		300	IN	A	$IP_ADDRESS
mail.domain2.com.	300	IN	A	$IP_ADDRESS

In the above snippets, $IP_ADDRESS should be the actual IPv4 address of the webserver. It should be noted that, in this setup, the web stack and the mail stack reside on the same physical host, so the IP is the same for both stacks.

Apache (webserver) configurations

As mentioned above, I keep the web stack and mail stack separate in terms of DNS. For the LetsEncrypt certificates (covered in the next section), though, I use the same certificate for both stacks. I do so by generating the cert for both the main domain and the ‘mail’ subdomain. In order for this to work, I make the ‘mail’ subdomain a ServerAlias in the Apache vhost configurations:

# grep -e 'ServerName \|ServerAlias ' www.domain1.com.conf 
	ServerName domain1.com
	ServerAlias www.domain1.com mail.domain1.com

# grep -e 'ServerName \|ServerAlias ' www.domain2.com.conf 
	ServerName domain2.com
	ServerAlias www.domain2.com mail.domain2.com

This allows the verification of the ‘mail’ subdomain to be done via the main URL instead of requiring a separate public-facing site directory for it.

LetsEncrypt SSL certificates

LetsEncrypt is a non-profit certificate authority (CA) that provides X.509 (TLS) certificates free-of-charge. The issued certificates are only valid for 90 days, which encourages automated processes to handle renewals. The recommended method is to use the certbot tool for renewals, and there are many plugins available that provide integration with various webservers. Though I run a combination of Apache and NGINX, I prefer to not have certbot directly interact with them. Rather, I choose to rely on certbot solely for the certificate generation & renewal, and to handle the installation thereof via other means. For this tutorial, I will use the ‘certonly’ option with the webroot plugin:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/$DOMAIN/$HOST/htdocs/ --domains $DOMAIN,$DOMAIN,$DOMAIN

In the code snippet above, you will replace $DOMAIN with the actual domain, and $HOST with the subdomain. So, for our two hypothetical domains, the commands translate as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,mail.domain1.com

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain2.com/www/htdocs/ --domains domain2.com,www.domain2.com,mail.domain2.com

The webroot plugin will create a temporary file under ${webroot-path}/.well-known/acme-challenge/ and then check that file via HTTP in order to validate the server. Make sure that the directory is publicly accessible or else the validation will fail. Once certbot validates the listed domains—in this setup, the ‘www’ and ‘mail’ subdomains are just aliases to the primary domain (see the BIND and Apache configurations sections above)—it will generate the SSL certificates and place them under /etc/letsencrypt/live/domain1.com/ and /etc/letsencrypt/live/domain2.com/, respectively. There will be two files for each certificate:

  • fullchain.pem –> the public certificate
  • privkey.pem –> the private key for the certificate

Though these certificates are often used in conjunction with the web stack, we are going to use them for securing the mail stack as well.

Dovecot (IMAP) configurations

Now that we have the certificates for each domain, we’ll start by securing the IMAP server (i.e. Dovecot) so that the users’ Mail User Agent (MUA, or more colloquially, “email client” [like Thunderbird or Outlook]) will no longer require a security exception due to a domain mismatch. Adding the domain-specific SSL certificate to Dovecot is a straightforward process that only requires two directives per domain. For domain1.com and domain2.com, add the following lines to /etc/dovecot/conf.d/10-ssl.conf:

local_name mail.domain1.com {
  ssl_cert = </etc/letsencrypt/live/domain1.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain1.com/privkey.pem
}

local_name mail.domain2.com {
  ssl_cert = </etc/letsencrypt/live/domain2.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain2.com/privkey.pem
}

Those code blocks can be copied and pasted for any additional virtual hosts or domains that are needed. As with any configuration change, make sure to restart the application in order to make the changes active:

/etc/init.d/dovecot restart

Postfix (SMTP) configurations

Configuring Dovecot to use the SSL certificate for securing the IMAP connection from the user’s email client to the server is only one part of the process—namely the connection when the user is retrieving mails from the server. This next part will use the same certificate to secure the SMTP connection (via the Postfix SMTP server) for sending mails.

The first step is to create a file that will be used for mapping each certificate to its respective domain. Postfix can handle this correlation via Server Name Indication (SNI), which is an extension of the TLS protocol that indicates the hostname of the server at the beginning of the handshake process. Though there is no naming requirement for this map file, I chose to create it as /etc/postfix/vmail_ssl. The format of the file is:

$DOMAIN   $PRIVATE_KEY   $PUBLIC_KEY_CHAIN

So for our example of domain1.com and domain2.com, the file would consist of the following entries:

mail.domain1.com /etc/letsencrypt/live/domain1.com/privkey.pem /etc/letsencrypt/live/domain1.com/fullchain.pem

mail.domain2.com /etc/letsencrypt/live/domain2.com/privkey.pem /etc/letsencrypt/live/domain2.com/fullchain.pem

Though this file is in plaintext, Postfix doesn’t understand the mapping in this format. Instead, a base64-encoded Berkeley DB file needs to be used. Thankfully, Postfix makes creating such a file very easy via the postmap utility. Once you have created and populated your /etc/postfix/vmail_ssl file with the entries for each domain, issue the following command:

# postmap -F hash:/etc/postfix/vmail_ssl

which will create the Berkeley DB file (named vmail_ssl.db) in the same directory:

# find /etc/postfix/ -type f -iname '*vmail_ssl*'
/etc/postfix/vmail_ssl.db
/etc/postfix/vmail_ssl

# file /etc/postfix/vmail_ssl
/etc/postfix/vmail_ssl: ASCII text

# file /etc/postfix/vmail_ssl.db 
/etc/postfix/vmail_ssl.db: Berkeley DB (Hash, version 10, native byte-order)

Now that we have created the mapping table, we have to configure Postfix to use it for SMTP connections. It’s acceptable to have both the SNI-mapped certificates AND a generic SSL certificate as the default (for when a domain isn’t listed in the mapping table). Postfix can have both directives specified simultaneously. To do so, the following directives need to be added to /etc/postfix/main.cf (the comments explain both sets of directives):

## Default SSL cert for SMTP if SNI is not enabled
smtpd_tls_cert_file = /etc/ssl/mail/server.pem
smtpd_tls_key_file = /etc/ssl/mail/server.key

## Mappings for SMTP SSL certs when SNI is enabled
tls_server_sni_maps = hash:/etc/postfix/vmail_ssl

After making those modifications to Postfix’s main configuration file, it is required to restart it:

/etc/init.d/postfix restart

That’s it! Now the full mail stack is secured using domain-specific SSL certificates for both IMAP and SMTP connections. The remaining sections below will explain some maintenance-related procedures such as handling the LetsEncrypt certificate renewals & updating the mappings in Postfix automatically, as well as verifying it’s all working as intended (and some troubleshooting tips in case it’s not). 🙂

Automatic renewal (cron) configurations

As mentioned in the LetsEncrypt section above, the certificates that they issue are only valid for a period of 90 days. One of the reasons for the relatively short validity period is to encourage automation when it comes to renewing them. I choose to handle the renewals automatically via cron:

# tail -n 4 /var/spool/cron/crontabs/root 

## LetsEncrypt certificate renewals on first of each month
## See /etc/letsencrypt/renewal-hooks/post/ for Postfix & Apache hooks 
0  2  1  *  *   /usr/bin/certbot renew --quiet

This cron entry instructs LetsEncrypt’s certbot to check the validity of ALL certificates at 02:00 (server time) on the first of every month (if that format is unfamiliar to you, see Wikipedia’s article on cron). The renew subcommand will automatically generate a new certificate for any found to expire within the next 30 days, and the quiet option will silence any output except for errors, which is appropriate for use with a cron job.

That’s the procedure for renewing the certificate automatically, but what about then automatically updating the appropriate stack configurations—in particular, Postfix’s vmail_ssl mappings table (and Apache, but that’s outside the scope of this tutorial)? If the certificate is renewed, but it is not updated in Postfix’s hash table, there will be a mismatch error. As mentioned in the comment on the cron entry, I chose to handle those configuration updates automatically via certbot’s ‘renewal hooks’, which can be found under /etc/letsencrypt/renewal-hooks/. In this case, the configuration updates need to happen after certificate renewal, so they are put under the post/ subdirectory.

I have two scripts that run after a certificate renewal, but only the 01_postfix_smtp_ssl.sh one is applicable for the mail stack:

# ls /etc/letsencrypt/renewal-hooks/post/
01_postfix_smtp_ssl.sh  02_apache.sh

# cat /etc/letsencrypt/renewal-hooks/post/01_postfix_smtp_ssl.sh 
#!/bin/bash
/usr/sbin/postmap -F hash:/etc/postfix/vmail_ssl
/etc/init.d/postfix restart
exit 0

The simple script issues the same postmap command from the ‘Postfix (SMTP) configurations‘ section above, and then restarts Postfix. If everything goes smoothly, it will exit cleanly (‘exit 0’). The script ensures that the new certificate is immediately applied to the Postfix configuration so that there aren’t validation errors after the automated renewal process.

Verification of the certificates

If everything went according to plan, valid SSL certificates should be in place for both mail.domain1.com and mail.domain2.com. Like any good engineer, though, we don’t want to just assume that it’s working as intended. So… we should test it! You could just open an email client of your choice and view the certificates for IMAP and SMTP connections. Personally, though, I prefer using terminal-based utilities as I find them to be more efficient. In this case, we can use the openssl command for connecting to each domain as a test, and the basic syntax is:

For SMTP:

openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp

For IMAP:

openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com

These commands will output a lot of information including the full public certificate, the issuing authority (LetsEncrypt), handshake details, SSL session details, and so on. If you’re interested in all of those details, feel free to issue the commands as they are above (obviously swapping out the actual domains and the ports that you use for SMTP and IMAP). If, however, you simply want to confirm that the certificates are valid, you can pipe the commands to grep in order to limit the output:

For SMTP:

$ openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
250 CHUNKING
subject=CN = domain1.com
Verify return code: 0 (ok)

For IMAP:

$ openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
subject=CN = domain1.com
Verify return code: 0 (ok)

If you see output similar to what’s above, then everything is working as it should be. In particular, you want to make sure that references to the ‘CN’ match the domain, and that you see a ‘Verify return code:’ of 0 (ok) Pat yourself on the back and grab your beverage of choice to celebrate a job well done. 🙂

Additional information

If you have already been using a domain for a website or other service, chances are that you have already generated a LetsEncrypt SSL certificate for it. Thankfully LetsEncrypt makes it easy to append a new subdomain to an existing certificate instead of having to generate a completely separate one for the ‘mail’ subdomain used in this guide (e.g. mail.domain1.com).

The first step is to find the certificate that you want to modify (in this case, domain1.com) and see which subdomains are covered under it. This can be accomplished using the certbot certificates command. The output will look something like this:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com
   Expiry Date: 2021-05-02 06:03:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

The important part is the list of subdomains on the Domains: line because you need to reference ALL of them when using the --expand flag that follows. Using the output from above, the command would be constructed as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,staging.domain1.com,mail.domain1.com --expand

If certbot indicates that the new certificate has been generated without any errors, you can check it again using the certbot certificates command from above and validate that now the ‘mail’ subdomain is listed as well:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com mail.domain1.com
   Expiry Date: 2021-05-14 06:05:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

Troubleshooting

I can’t anticipate the full gamut of problems that could potentially arise when going through this guide, but I will try to cover some common pitfalls here. If you run into a problem, feel free to comment and I will try to help you through it.

>>> Postfix error about the table hash:

If Postfix won’t start after the modifications from the sections above, and you see a line like this in the mail logs:

[postfix/smtpd] warning: table hash:/etc/postfix/vmail_ssl.db: key mail.domain1.com: malformed BASE64 value: /etc/letsencrypt/live/domain1

then the problem stems from running postmap without the -F flag. Try it again with that flag: postmap -F hash:/etc/postfix/vmail_ssl which should create a syntactically correct hash table, allowing Postfix to properly start up.

March 14 2021

Gentoo AMD64 Handbook "Preparing the disks" section reworked

Andreas K. Hüttel (dilfridge) March 14, 2021, 9:55

♦Since the text was becoming more and more outdated and also more and more convoluted, I have completely reworked the "Preparing the disks" section of the Gentoo AMD64 handbook. 

  • Since fdisk supports GUID partition tables (GPT) for a long time now, references to parted have been dropped.
  • The text restricts itself now to the combinations 1) UEFI boot and GPT and 2) BIOS / legacy boot and MBR. While mixing and matching is here certainly possible, we should treat it out of the scope of the manual.
  • Hopefully the terminology regarding the boot partition, UEFI system partition, and BIOS boot partition is more clear now (it was horribly mixed up before).

Please proofread and check for mistakes! I'll drop the "work in progress" label in a few days if nothing comes up.

Since the text was becoming more and more outdated and also more and more convoluted, I have completely reworked the "Preparing the disks" section of the Gentoo AMD64 handbook

  • Since fdisk supports GUID partition tables (GPT) for a long time now, references to parted have been dropped.
  • The text restricts itself now to the combinations 1) UEFI boot and GPT and 2) BIOS / legacy boot and MBR. While mixing and matching is here certainly possible, we should treat it out of the scope of the manual.
  • Hopefully the terminology regarding the boot partition, UEFI system partition, and BIOS boot partition is more clear now (it was horribly mixed up before).

Please proofread and check for mistakes! I'll drop the "work in progress" label in a few days if nothing comes up.

March 03 2021

Moving commits between independent git histories

Michał Górny (mgorny) March 03, 2021, 17:55

PyPy is an alternative Python implementation. While it does replace a large part of the interpreter, a large part of the standard library is shared with CPython. As a result, PyPy is frequently affected by the same vulnerabilities as CPython, and we have to backport security fixes to it.

Backporting security fixes inside CPython is relatively easy. All main Python branches are in a single repository, so it’s just a matter of cherry-picking the commits. Normally, you can easily move patches between two related git repositories using git-style patches but this isn’t going to work for two repositories with unrelated histories.

Does this mean manually patching PyPy and rewriting commit messages by hand? Luckily, there’s a relatively simple git am trick that can help you avoid that.

Roughly, the idea is to:

1. Create a git-format-patch of the change to backport.

2. Attempt to apply the change via git am — it will fail and your repository will be left in middle of an am session.

3. Apply the change via patch.

4. git add the changes.

5. Finally, call git am --continue to finish, wrapping your changes in the original commit metadata.

For example, let’s try backporting CVE-2021-23336 (parameter cloaking) fix:

First, grab the relevant patch from the CPython repository:

$ git format-patch -1 d0d4d30882fe3ab9b1badbecf5d15d94326fd13e
0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch

Then, inside the local clone of a random PyPy git mirror:

$ git am -3 ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
error: sha1 information is lacking or useless (Doc/library/cgi.rst).
error: could not build fake ancestor
Patch failed at 0001 bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Now enter the directory that stdlib resides in, and apply the patch manually, skipping any missing files:

$ patch -p2 < ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
can't find file to patch at input line 39
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|From d0d4d30882fe3ab9b1badbecf5d15d94326fd13e Mon Sep 17 00:00:00 2001
|From: Senthil Kumaran 
|Date: Mon, 15 Feb 2021 10:34:14 -0800
|Subject: [PATCH] [3.7] bpo-42967: only use '&' as a query string separator
| (GH-24297)  (GH-24531)
|MIME-Version: 1.0
|Content-Type: text/plain; charset=UTF-8
|Content-Transfer-Encoding: 8bit
|
|bpo-42967: [security] Address a web cache-poisoning issue reported in
|urllib.parse.parse_qsl().
|
|urllib.parse will only us "&" as query string separator by default
|instead of both ";" and "&" as allowed in earlier versions. An optional
|argument seperator with default value "&" is added to specify the
|separator.
|
|Co-authored-by: Éric Araujo 
|Co-authored-by: Ken Jin 
|Co-authored-by: Adam Goldschmidt 
|(cherry picked from commit fcbe0cb04d35189401c0c880ebfb4311e952d776)
|---
| Doc/library/cgi.rst                           |  9 ++-
| Doc/library/urllib.parse.rst                  | 23 ++++++-
| Doc/whatsnew/3.6.rst                          | 13 ++++
| Doc/whatsnew/3.7.rst                          | 13 ++++
| Lib/cgi.py                                    | 23 ++++---
| Lib/test/test_cgi.py                          | 29 ++++++--
| Lib/test/test_urlparse.py                     | 68 +++++++++++++------
| Lib/urllib/parse.py                           | 19 ++++--
| .../2021-02-14-15-59-16.bpo-42967.YApqDS.rst  |  1 +
| 9 files changed, 152 insertions(+), 46 deletions(-)
| create mode 100644 Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst
|
|diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst
|index 0b1aead9dd..f0ec7e8cc6 100644
|--- a/Doc/library/cgi.rst
|+++ b/Doc/library/cgi.rst
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
3 out of 3 hunks ignored
[...]
patching file cgi.py
patching file test/test_cgi.py
patching file test/test_urlparse.py
patching file urllib/parse.py
patching file NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst

Adjust the changes as appropriate:

$ rm -r NEWS.d/
$ git status
HEAD detached from release-pypy3.7-v7.3.3
You are in the middle of an am session.
  (fix conflicts and then run "git am --continue")
  (use "git am --skip" to skip this patch)
  (use "git am --abort" to restore the original branch)

Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git restore ..." to discard changes in working directory)
	modified:   cgi.py
	modified:   test/test_cgi.py
	modified:   test/test_urlparse.py
	modified:   urllib/parse.py

no changes added to commit (use "git add" and/or "git commit -a")
$ git add cgi.py test/test_cgi.py test/test_urlparse.py urllib/parse.py

And finally let git am commit the changes for you:

$ git am --continue
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)

PyPy is an alternative Python implementation. While it does replace a large part of the interpreter, a large part of the standard library is shared with CPython. As a result, PyPy is frequently affected by the same vulnerabilities as CPython, and we have to backport security fixes to it.

Backporting security fixes inside CPython is relatively easy. All main Python branches are in a single repository, so it’s just a matter of cherry-picking the commits. Normally, you can easily move patches between two related git repositories using git-style patches but this isn’t going to work for two repositories with unrelated histories.

Does this mean manually patching PyPy and rewriting commit messages by hand? Luckily, there’s a relatively simple git am trick that can help you avoid that.

Roughly, the idea is to:

1. Create a git-format-patch of the change to backport.

2. Attempt to apply the change via git am — it will fail and your repository will be left in middle of an am session.

3. Apply the change via patch.

4. git add the changes.

5. Finally, call git am --continue to finish, wrapping your changes in the original commit metadata.

For example, let’s try backporting CVE-2021-23336 (parameter cloaking) fix:

First, grab the relevant patch from the CPython repository:

$ git format-patch -1 d0d4d30882fe3ab9b1badbecf5d15d94326fd13e
0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch

Then, inside the local clone of a random PyPy git mirror:

$ git am -3 ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
error: sha1 information is lacking or useless (Doc/library/cgi.rst).
error: could not build fake ancestor
Patch failed at 0001 bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Now enter the directory that stdlib resides in, and apply the patch manually, skipping any missing files:

$ patch -p2 < ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
can't find file to patch at input line 39
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|From d0d4d30882fe3ab9b1badbecf5d15d94326fd13e Mon Sep 17 00:00:00 2001
|From: Senthil Kumaran 
|Date: Mon, 15 Feb 2021 10:34:14 -0800
|Subject: [PATCH] [3.7] bpo-42967: only use '&' as a query string separator
| (GH-24297)  (GH-24531)
|MIME-Version: 1.0
|Content-Type: text/plain; charset=UTF-8
|Content-Transfer-Encoding: 8bit
|
|bpo-42967: [security] Address a web cache-poisoning issue reported in
|urllib.parse.parse_qsl().
|
|urllib.parse will only us "&" as query string separator by default
|instead of both ";" and "&" as allowed in earlier versions. An optional
|argument seperator with default value "&" is added to specify the
|separator.
|
|Co-authored-by: Éric Araujo 
|Co-authored-by: Ken Jin 
|Co-authored-by: Adam Goldschmidt 
|(cherry picked from commit fcbe0cb04d35189401c0c880ebfb4311e952d776)
|---
| Doc/library/cgi.rst                           |  9 ++-
| Doc/library/urllib.parse.rst                  | 23 ++++++-
| Doc/whatsnew/3.6.rst                          | 13 ++++
| Doc/whatsnew/3.7.rst                          | 13 ++++
| Lib/cgi.py                                    | 23 ++++---
| Lib/test/test_cgi.py                          | 29 ++++++--
| Lib/test/test_urlparse.py                     | 68 +++++++++++++------
| Lib/urllib/parse.py                           | 19 ++++--
| .../2021-02-14-15-59-16.bpo-42967.YApqDS.rst  |  1 +
| 9 files changed, 152 insertions(+), 46 deletions(-)
| create mode 100644 Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst
|
|diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst
|index 0b1aead9dd..f0ec7e8cc6 100644
|--- a/Doc/library/cgi.rst
|+++ b/Doc/library/cgi.rst
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
3 out of 3 hunks ignored
[...]
patching file cgi.py
patching file test/test_cgi.py
patching file test/test_urlparse.py
patching file urllib/parse.py
patching file NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst

Adjust the changes as appropriate:

$ rm -r NEWS.d/
$ git status
HEAD detached from release-pypy3.7-v7.3.3
You are in the middle of an am session.
  (fix conflicts and then run "git am --continue")
  (use "git am --skip" to skip this patch)
  (use "git am --abort" to restore the original branch)

Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git restore ..." to discard changes in working directory)
	modified:   cgi.py
	modified:   test/test_cgi.py
	modified:   test/test_urlparse.py
	modified:   urllib/parse.py

no changes added to commit (use "git add" and/or "git commit -a")
$ git add cgi.py test/test_cgi.py test/test_urlparse.py urllib/parse.py

And finally let git am commit the changes for you:

$ git am --continue
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)

February 23 2021

Why not rely on app developer to handle security?

Michał Górny (mgorny) February 23, 2021, 20:16

One of the comments to the The modern packager’s security nightmare post posed a very important question: why is it bad to depend on the app developer to address security issues? In fact, I believe it is important enough to justify a whole post discussing the problem. To clarify, the wider context is bundling dependencies, i.e. relying on the application developer to ensure that all the dependencies included with the application to be free of vulnerabilities.

In my opinion, the root of security in open source software is widely understood auditing. Since the code is public, everyone can read it, analyze it, test it. However, with a typical system install including thousands of packages from hundreds of different upstreams, it is really impossible even for large companies (not to mention individuals) to be able to audit all that code. Instead, we assume that with large enough number of eyes looking at the code, all vulnerabilities will eventually be found and published.

On top of auditing we add trust. Today, CVE authorities are at the root of our vulnerability trust. We trust them to reliably publish reports of vulnerabilities found in various packages. However, once again we can’t expect users to manually make sure that the huge number of the packages they are running are free of vulnerabilities. Instead, the trust is hierarchically moved down to software authors and distributions.

Both software authors and distribution packagers share a common goal — ensuring that their end users are running working, secure software. Why do I believe then that the user’s trust is better placed in distribution packagers than in software authors? I am going to explain this in three points.

How many entities do you trust?

The following graph depicts a fragment of dependency chain between a few C libraries and end-user programs. The right-hand side depicts leaf packages, those intended for the user. On the left side, you can see some of the libraries they use, or that their dependent libraries use. All of this stems from OpenSSL.

Now imagine that a vulnerability is found in OpenSSL. Let’s discuss what would happen in two worlds: one where the distribution is responsible for ensuring the security of all the packages, and another one where upstreams bundle dependencies and are expected to handle it.

In the distribution world, the distribution maintainers are expected to ensure that every node of that dependency graph is secure. If a vulnerability is found in OpenSSL, it is their responsibility to realize that and update the vulnerable package. The end users effectively have to trust distribution maintainers to keep their systems secure.

In the bundled dependency world, the maintainer of every successive node needs to ensure the security of its dependencies. First, the maintainers of cURL, Transmission, Tor, Synergy and Qt5 need to realize that OpenSSL has a new vulnerability, update their bundled versions and make new releases of their software. Then, the maintainers of poppler, cmake, qemu and Transmission (again) need to update their bundled versions of cURL to transitively avoid the vulnerable OpenSSL version. Same goes for the maintainers of Synergy (again) and KeepAssXC that have to update their bundled Qt5. Finally, the maintainers of Inkscape and XeTeX need to update their bundled poppler.

Even if we disregard the amount of work involved and the resulting slowdown in deploying the fix, the end user needs to trust 11 different entities to fix their respective software packages not to ship (transitively) a vulnerable version of OpenSSL. And the most absurd thing is, they will nevertheless need to trust their distribution vendor to actually ship all these updated packages.

I think that we can mostly agree that trusting a single entity provides much smaller attack surface than trusting tens or hundreds of different entities.

The above example has been placed in the C world of relatively few relatively large libraries. The standard C library is pretty fat, the standard C++ library is much fatter. A typical examples of the C approach are libraries like libgcrypt — a single library providing a lot of cryptographic primitives such as ciphers along with modes, hashes, MACs, RNGs and related functions — or OpenSSL — providing both crypto primitives and a TLS implementation on top of it. A common Rust/Cargo approach (though things are not always done this way) is to have a separate crate for every algorithm, in every variant. I’m not saying this is wrong — in fact, I actually like the idea of small libraries that do one thing well. However, it multiplies the aforementioned problem thousandfold.

The bus factor

The second part of the problem is the bus factor.

A typical ‘original’ Linux distribution involves a few hundred developers working together on software. There are much smaller distributions (or forks) but they are generally building on something bigger. These distributions generally have dedicated teams in place to handle security, as well as mechanisms to help them with their work.

For example, in Gentoo security issues can be usually tackled from two ends. On one end, there’s the package maintainer who takes care of daily maintenance tasks and usually notices vulnerability fixes through new releases. On the other end, there’s a dedicated Security team whose members are responsible for monitoring new CVEs. Most of the time, these people work together to resolve security issues, with the maintainer having specific knowledge of the package in question and the Security team sharing general security experience and monitoring the process.

Should any maintainer be away or otherwise unable to fix the vulnerability quickly, the Security team can chime in and take care of whatever needs to be done. These people are specifically focused on that one job, and this means that the chances of things going sideways are comparatively small. Even if the whole distribution were to suddenly disappear, the users have a good chance of noticing that.

Besides a few very large software packages, most of the software projects are small. It is not uncommon for a software package to be maintained by a single person. Now, how many dependencies can a single person or a small team effectively maintain? Even with the best interest at heart, it is unlikely that a software developer whose primary goal is to work on code of the particular project can be reasonably expected to share the same level of dedication and experience regarding the project’s dependencies as dedicated distribution maintainers who are doing full-time maintenance work on them.

Even if we could reasonably assume that we can trust all upstreams to do their best to ensure that their dependencies are not vulnerable, it is inevitable that some of them will not be able to handle this timely. In fact, some projects suddenly become abandoned and then vulnerabilities are not handled at all. Now, the problem is not only that it might happen. The problem is how to detect the problem early, and how to deal with it. Can you be reasonable expected to monitor hundreds of upstreams for activity? Again, the responsibility falls on distribution developers who would have to resolve these issues independently.

How much testing can you perform?

The third point is somewhat less focused on security, and more on bugs in general. Bundling dependencies not only defers handling security in packages to the application developers but also all other upgrades. The comment author argues: I want to ship software that I know works, and I can only do that if I know what my dependencies actually are so I can test against them. That’s a valid point but there’s a but: how many real life scenarios can you actually test?

Let’s start with the most basic stuff. Your CI most likely runs on a 64-bit x86 system. Some projects test more but it’s still a very limited hardware set to test. If one of your dependencies is broken on non-x86 architecture, your testing is unlikely to catch that. Even if the authors of that dependency are informed about the problem and release a fixed version, you won’t know that the upgrade is necessary unless someone reports the problem to you (and you may actually have a problem attributing the issue to the particular dependency).

In reality, things aren’t always this good. Not all upstreams release fixes quickly. Distribution packagers sometimes have to backport or even apply custom patches to make things work. If packages bundle dependencies, it is not sufficient to apply the fix at the root — it needs to be applied to all packages bundling the dependency. In the end, it is even possible that different upstreams will start applying different patches to the same dependencies to resolve the same problem independently reported to all of them. This means more maintenance work for you, and a maintenance nightmare for distributions.

There are also other kinds of issues that CIs often don’t catch. ‘Unexpected’ system setup, different locale, additional packages installed (Python is sometimes quite fragile to that). Your testing can’t really predict all possible scenarios and protect against them. Pinning to a dependency that you know to be good for you does not actually guarantee that it will be good for all your end users. By blocking upgrades, you may actually expose them to bugs that were already known and fixed.

Summary

Bundling dependencies is bad. You can reasonably assume that you’re going to do a good job at maintaining your package and keeping its bundled dependencies secure. However, you can’t guarantee that the maintainers of other packages involved will do the same. And you can’t reasonably expect the end user to place trust in the security of his system to hundreds of different people. The stakes are high and the risk is huge.

The number of entities involved is just too great. You can’t expect anyone to reasonably monitor them, and with many projects having no more than a single developer, you can’t guarantee that the fixes will be handled promptly and that the package in question will not be abandoned. A single package in middle of a dependency chain could effectively render all its reverse dependencies vulnerable, and multiply the work of their maintainers who have to locally fix the problem.

In the end, the vast majority of Linux users need to trust their distribution to ensure that the packages shipped to them are not vulnerable. While you might think that letting you handle security makes things easier for us, it doesn’t. We still need to monitor all the packages and their dependencies. The main difference is that we can fix it in one place, while upstreams have to fix it everywhere. And while they do, we need to ensure that all of them actually did that, and often it is hard to even find all the bundled dependencies (including inline code copied from other projects) as there are no widely followed standards for doing this.

So even if we ignored all the other technical downsides to bundled dependencies, the sum total of work needed to keep packages bundling it secure is much greater than the cost of unbundling the dependencies.

You should ask yourself the following question: do you really want to be responsible for all these dependencies? As a software developer, you want to focus on writing code. You want to make sure that your application works. There are other people who are ready and willing to take care of the ecosystem your software is going to run in. Fit your program into the environment, instead of building an entirely new world for it. When you do that, you’ve effectively replicating a subset of a distribution. Expecting every single application developer to do the necessary work (and to have the necessary knowledge) does not scale.

One of the comments to the The modern packager’s security nightmare post posed a very important question: why is it bad to depend on the app developer to address security issues? In fact, I believe it is important enough to justify a whole post discussing the problem. To clarify, the wider context is bundling dependencies, i.e. relying on the application developer to ensure that all the dependencies included with the application to be free of vulnerabilities.

In my opinion, the root of security in open source software is widely understood auditing. Since the code is public, everyone can read it, analyze it, test it. However, with a typical system install including thousands of packages from hundreds of different upstreams, it is really impossible even for large companies (not to mention individuals) to be able to audit all that code. Instead, we assume that with large enough number of eyes looking at the code, all vulnerabilities will eventually be found and published.

On top of auditing we add trust. Today, CVE authorities are at the root of our vulnerability trust. We trust them to reliably publish reports of vulnerabilities found in various packages. However, once again we can’t expect users to manually make sure that the huge number of the packages they are running are free of vulnerabilities. Instead, the trust is hierarchically moved down to software authors and distributions.

Both software authors and distribution packagers share a common goal — ensuring that their end users are running working, secure software. Why do I believe then that the user’s trust is better placed in distribution packagers than in software authors? I am going to explain this in three points.

How many entities do you trust?

The following graph depicts a fragment of dependency chain between a few C libraries and end-user programs. The right-hand side depicts leaf packages, those intended for the user. On the left side, you can see some of the libraries they use, or that their dependent libraries use. All of this stems from OpenSSL.

A graph depicting dependencies between a few C libraries and end-user programs

Now imagine that a vulnerability is found in OpenSSL. Let’s discuss what would happen in two worlds: one where the distribution is responsible for ensuring the security of all the packages, and another one where upstreams bundle dependencies and are expected to handle it.

In the distribution world, the distribution maintainers are expected to ensure that every node of that dependency graph is secure. If a vulnerability is found in OpenSSL, it is their responsibility to realize that and update the vulnerable package. The end users effectively have to trust distribution maintainers to keep their systems secure.

In the bundled dependency world, the maintainer of every successive node needs to ensure the security of its dependencies. First, the maintainers of cURL, Transmission, Tor, Synergy and Qt5 need to realize that OpenSSL has a new vulnerability, update their bundled versions and make new releases of their software. Then, the maintainers of poppler, cmake, qemu and Transmission (again) need to update their bundled versions of cURL to transitively avoid the vulnerable OpenSSL version. Same goes for the maintainers of Synergy (again) and KeepAssXC that have to update their bundled Qt5. Finally, the maintainers of Inkscape and XeTeX need to update their bundled poppler.

Even if we disregard the amount of work involved and the resulting slowdown in deploying the fix, the end user needs to trust 11 different entities to fix their respective software packages not to ship (transitively) a vulnerable version of OpenSSL. And the most absurd thing is, they will nevertheless need to trust their distribution vendor to actually ship all these updated packages.

I think that we can mostly agree that trusting a single entity provides much smaller attack surface than trusting tens or hundreds of different entities.

The above example has been placed in the C world of relatively few relatively large libraries. The standard C library is pretty fat, the standard C++ library is much fatter. A typical examples of the C approach are libraries like libgcrypt — a single library providing a lot of cryptographic primitives such as ciphers along with modes, hashes, MACs, RNGs and related functions — or OpenSSL — providing both crypto primitives and a TLS implementation on top of it. A common Rust/Cargo approach (though things are not always done this way) is to have a separate crate for every algorithm, in every variant. I’m not saying this is wrong — in fact, I actually like the idea of small libraries that do one thing well. However, it multiplies the aforementioned problem thousandfold.

The bus factor

The second part of the problem is the bus factor.

A typical ‘original’ Linux distribution involves a few hundred developers working together on software. There are much smaller distributions (or forks) but they are generally building on something bigger. These distributions generally have dedicated teams in place to handle security, as well as mechanisms to help them with their work.

For example, in Gentoo security issues can be usually tackled from two ends. On one end, there’s the package maintainer who takes care of daily maintenance tasks and usually notices vulnerability fixes through new releases. On the other end, there’s a dedicated Security team whose members are responsible for monitoring new CVEs. Most of the time, these people work together to resolve security issues, with the maintainer having specific knowledge of the package in question and the Security team sharing general security experience and monitoring the process.

Should any maintainer be away or otherwise unable to fix the vulnerability quickly, the Security team can chime in and take care of whatever needs to be done. These people are specifically focused on that one job, and this means that the chances of things going sideways are comparatively small. Even if the whole distribution were to suddenly disappear, the users have a good chance of noticing that.

Besides a few very large software packages, most of the software projects are small. It is not uncommon for a software package to be maintained by a single person. Now, how many dependencies can a single person or a small team effectively maintain? Even with the best interest at heart, it is unlikely that a software developer whose primary goal is to work on code of the particular project can be reasonably expected to share the same level of dedication and experience regarding the project’s dependencies as dedicated distribution maintainers who are doing full-time maintenance work on them.

Even if we could reasonably assume that we can trust all upstreams to do their best to ensure that their dependencies are not vulnerable, it is inevitable that some of them will not be able to handle this timely. In fact, some projects suddenly become abandoned and then vulnerabilities are not handled at all. Now, the problem is not only that it might happen. The problem is how to detect the problem early, and how to deal with it. Can you be reasonable expected to monitor hundreds of upstreams for activity? Again, the responsibility falls on distribution developers who would have to resolve these issues independently.

How much testing can you perform?

The third point is somewhat less focused on security, and more on bugs in general. Bundling dependencies not only defers handling security in packages to the application developers but also all other upgrades. The comment author argues: I want to ship software that I know works, and I can only do that if I know what my dependencies actually are so I can test against them. That’s a valid point but there’s a but: how many real life scenarios can you actually test?

Let’s start with the most basic stuff. Your CI most likely runs on a 64-bit x86 system. Some projects test more but it’s still a very limited hardware set to test. If one of your dependencies is broken on non-x86 architecture, your testing is unlikely to catch that. Even if the authors of that dependency are informed about the problem and release a fixed version, you won’t know that the upgrade is necessary unless someone reports the problem to you (and you may actually have a problem attributing the issue to the particular dependency).

In reality, things aren’t always this good. Not all upstreams release fixes quickly. Distribution packagers sometimes have to backport or even apply custom patches to make things work. If packages bundle dependencies, it is not sufficient to apply the fix at the root — it needs to be applied to all packages bundling the dependency. In the end, it is even possible that different upstreams will start applying different patches to the same dependencies to resolve the same problem independently reported to all of them. This means more maintenance work for you, and a maintenance nightmare for distributions.

There are also other kinds of issues that CIs often don’t catch. ‘Unexpected’ system setup, different locale, additional packages installed (Python is sometimes quite fragile to that). Your testing can’t really predict all possible scenarios and protect against them. Pinning to a dependency that you know to be good for you does not actually guarantee that it will be good for all your end users. By blocking upgrades, you may actually expose them to bugs that were already known and fixed.

Summary

Bundling dependencies is bad. You can reasonably assume that you’re going to do a good job at maintaining your package and keeping its bundled dependencies secure. However, you can’t guarantee that the maintainers of other packages involved will do the same. And you can’t reasonably expect the end user to place trust in the security of his system to hundreds of different people. The stakes are high and the risk is huge.

The number of entities involved is just too great. You can’t expect anyone to reasonably monitor them, and with many projects having no more than a single developer, you can’t guarantee that the fixes will be handled promptly and that the package in question will not be abandoned. A single package in middle of a dependency chain could effectively render all its reverse dependencies vulnerable, and multiply the work of their maintainers who have to locally fix the problem.

In the end, the vast majority of Linux users need to trust their distribution to ensure that the packages shipped to them are not vulnerable. While you might think that letting you handle security makes things easier for us, it doesn’t. We still need to monitor all the packages and their dependencies. The main difference is that we can fix it in one place, while upstreams have to fix it everywhere. And while they do, we need to ensure that all of them actually did that, and often it is hard to even find all the bundled dependencies (including inline code copied from other projects) as there are no widely followed standards for doing this.

So even if we ignored all the other technical downsides to bundled dependencies, the sum total of work needed to keep packages bundling it secure is much greater than the cost of unbundling the dependencies.

You should ask yourself the following question: do you really want to be responsible for all these dependencies? As a software developer, you want to focus on writing code. You want to make sure that your application works. There are other people who are ready and willing to take care of the ecosystem your software is going to run in. Fit your program into the environment, instead of building an entirely new world for it. When you do that, you’ve effectively replicating a subset of a distribution. Expecting every single application developer to do the necessary work (and to have the necessary knowledge) does not scale.

February 19 2021

The modern packager’s security nightmare

Michał Górny (mgorny) February 19, 2021, 22:50

One of the most important tasks of the distribution packager is to ensure that the software shipped to our users is free of security vulnerabilities. While finding and fixing the vulnerable code is usually considered upstream’s responsibility, the packager needs to ensure that all these fixes reach the end users ASAP. With the aid of central package management and dynamic linking, the Linux distributions have pretty much perfected the deployment of security fixes. Ideally, fixing a vulnerable dependency is as simple as patching a single shared library via the distribution’s automated update system.

Of course, this works only if the package in question is actually following good security practices. Over the years, many Linux distributions (at the very least, Debian, Fedora and Gentoo) have been fighting these bad practices with some success. However, today the times have changed. Today, for every 10 packages fixed, a completely new ecosystem emerges with the bad security practices at its central point. Go, Rust and to some extent Python are just a few examples of programming languages that have integrated the bad security practices into the very fabric of their existence, and recreated the same old problems in entirely new ways.

The root issue of bundling dependencies has been discussed many times before. The Gentoo Wiki explains why you should not bundle dependencies, and links to more material about it. I would like to take a bit wider approach, and discuss not only bundling (or vendoring) dependencies but also two closely relevant problems: static linking and pinning dependencies.

Static linking

In the simplest words, static linking means embedding your program’s dependencies directly into the program image. The term is generally used in contrast to dynamic linking (or dynamic loading) that keep the dependent libraries in separate files that are loaded at program’s startup (or runtime).

Why is static linking bad? The primary problem is that since they become an integral part of the program, they can not be easily replaced by another version. If it turns out that one of the libraries is vulnerable, you have to relink the whole program against the new version. This also implies that you need to have a system that keeps track of what library versions are used in individual programs.

While you might think that rebuilding a lot of packages is only a problem for source distributions, you are wrong. While indeed the users of source distributions could be impacted a lot, as their systems remain vulnerable for a long time needed to rebuild a lot of packages, a similar problem affects binary distributions. After all, the distributions need to rebuild all affected programs in order to fully ship the fix to their end users which also involves some delay.

Comparatively, shipping a new version of a shared library takes much less time and fixes all affected programs almost instantly (modulo the necessity of restarting them).

The extreme case of static linking is to distribute proprietary software that is statically linked to its dependencies. This is primarily done to ensure that the software can be run easily on a variety of systems without requiring the user to install its dependencies manually. However, this scenario is really a form of bundling dependencies, so it will be discussed in the respective section.

However, static linking has also been historically used for system programs that were meant to keep working even if their dependent libraries became broken.

In modern packages, static linking is used for another reason entirely — because they do not require the modern programming languages to have a stable ABI. The Go compiler does not need to be concerned about emitting code that would be binary compatible with the code coming from a previous version. It works around the problem by requiring you to rebuild everything every time the compiler is upgraded.

To follow the best practices, we strongly discourage static linking in C and its derivatives. However, we can’t do much about languages such as Go or Rust that put static linking at the core of their design and have time and again stated publicly that they will not switch to dynamic linking of dependencies.

Pinning dependencies

While static linking is bad, at least it provides a reasonably clear way for automatic updates (and therefore the propagation of vulnerability fixes) to happen, pinning dependencies means requiring a specific version of your program’s dependencies to be installed. While the exact results depend on the ecosystem and the exact way of pinning the dependency, generally it means that at least some users of your package will not be able to automatically update the dependencies to newer versions.

That might not seem that bad at first. However, it means that if a bug fix or — even more importantly — a vulnerability fix is released for the dependency, the users will not get it unless you update the pin and make a new release. And then, if somebody else pins your package, then that pin will also need to be updated and released. And the chain goes on. Not to mention what happens if some package just happens to indirectly pin to two different versions of the same dependency!

Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. However, all that has another underlying problem — the combination of not being concerned with API stability on upstream part, and not wishing to unnecessarily update working code (that uses deprecated API) on downstream part. Truth is, pinning makes this worse because it sweeps the problem under the carpet, and actively encourages people to develop their code against specific versions of their dependencies rather than against a stable public API. Hyrum’s Law in practice.

Dependency pinning can have really extreme consequences. Unless you make sure to update your pins often, you may one day find yourself having to take a sudden leap — because you have relied on a very old version of a dependency that is now known to be vulnerable, and in order to update it you suddenly have to rewrite a lot of code to follow the API changes. Long term, this approach simply does not scale anymore, the effort needed to keep things working grows exponentially.

We try hard to unpin the dependencies and test packages with the newest versions of them. However, often we end up discovering that the newer versions of dependencies simply are not compatible with the packages in question. Sadly, upstreams often either ignore reports of these incompatibilities or even are actively hostile to us for not following their pins.

Bundling/vendoring dependencies

Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used.

In open source software, bundling usually means either including the sources of your dependencies along with your program or making the build system fetch them automatically, and then building them along with the program. In closed source software, it usually means linking the program to its dependencies statically or including the dependency libraries along with the program.

The baseline problem is the same as with pinned dependencies — if one of them turns out to be buggy or vulnerable, the users need to wait for a new release to update the bundled dependency. In open source software or closed source software using dynamic libraries, the packager has at least a reasonable chance of replacing the problematic dependency or unbundling it entirely (i.e. forcing the system library). In statically linked closed source software, it is often impossible to even reliably determine what libraries were actually used, not to mention their exact versions. Your distribution can no longer reliably monitor security vulnerabilities; the trust is shifted to software vendors.

However, modern software sometimes takes a step further — and vendor modified dependencies. The horror of it! Now not only the packager needs to work to replace the library but often has to actually figure out what was changed compared to the original version, and rebase the changes. In worst cases, the code becomes disconnected from upstream to the point that the program author is no longer capable of updating the vendored dependency properly.

Sadly, this kind of vendoring is becoming more common with the rapid development happening these days. The cause is twofold. On one hand, downstream consumers find it easier to fork and patch a dependency than to work with upstreams. On the other hand, many upstreams are not really concerned with fixing bugs and feature requests that do not affect their own projects. Even if the fork is considered only as a stop-gap measure, it often takes a real lot of effort to push the changes upstream afterwards and re-synchronize the codebases.

We are strongly opposed to bundling dependencies. Whenever possible, we try to unbundle them — sometimes having to actually patch the build systems to reuse system libraries. However, this is a lot of work, and often it is not even possible because of custom patching, including the kind of patching that has been explicitly rejected upstream. To list a few examples — Mozilla products rely on SQLite 3 patches that collide with regular usage of this library, Rust bundles a fork of LLVM.

Summary

Static linking, dependency pinning and bundling are three bad practices that have serious impact on the time and effort needed to eliminate vulnerabilities from production systems. They can make the difference between being able to replace a vulnerable library within a few minutes and having to spend a lot of effort and time in locating multiple copies of the vulnerable library, patching and rebuilding all the software including them.

The major Linux distributions had policies against these practices for a very long time, and have been putting a lot of effort into eliminating them. Nevertheless, it feels more and more like Sisyphean task. While we have been able to successfully resolve these problems in many packages, whole new ecosystems were built on top of these bad practices — and it does not seem that upstreams care about fixing them at all.

New programming languages such as Go and Rust rely entirely on static linking, and there’s nothing we can do about it. Instead of packaging the dependencies and having programs use the newest versions, we just fetch the versions pinned by upstream and make big blobs out of it. And while upstreams brag how they magically resolved all security issues you could ever think of (entirely ignoring other classes of security issues than memory-related), we just hope that we won’t suddenly be caught with our pants down when a common pinned dependency of many packages turns out to be vulnerable.

Added 2021-04-27: Deadlock vulnerability through embedded app-emulation/containers-storage (CVE-2021-20291) is an example how a vulnerability in a single Go package requires hunting all affected ebuilds.

One of the most important tasks of the distribution packager is to ensure that the software shipped to our users is free of security vulnerabilities. While finding and fixing the vulnerable code is usually considered upstream’s responsibility, the packager needs to ensure that all these fixes reach the end users ASAP. With the aid of central package management and dynamic linking, the Linux distributions have pretty much perfected the deployment of security fixes. Ideally, fixing a vulnerable dependency is as simple as patching a single shared library via the distribution’s automated update system.

Of course, this works only if the package in question is actually following good security practices. Over the years, many Linux distributions (at the very least, Debian, Fedora and Gentoo) have been fighting these bad practices with some success. However, today the times have changed. Today, for every 10 packages fixed, a completely new ecosystem emerges with the bad security practices at its central point. Go, Rust and to some extent Python are just a few examples of programming languages that have integrated the bad security practices into the very fabric of their existence, and recreated the same old problems in entirely new ways.

The root issue of bundling dependencies has been discussed many times before. The Gentoo Wiki explains why you should not bundle dependencies, and links to more material about it. I would like to take a bit wider approach, and discuss not only bundling (or vendoring) dependencies but also two closely relevant problems: static linking and pinning dependencies.

Static linking

In the simplest words, static linking means embedding your program’s dependencies directly into the program image. The term is generally used in contrast to dynamic linking (or dynamic loading) that keep the dependent libraries in separate files that are loaded at program’s startup (or runtime).

Why is static linking bad? The primary problem is that since they become an integral part of the program, they can not be easily replaced by another version. If it turns out that one of the libraries is vulnerable, you have to relink the whole program against the new version. This also implies that you need to have a system that keeps track of what library versions are used in individual programs.

While you might think that rebuilding a lot of packages is only a problem for source distributions, you are wrong. While indeed the users of source distributions could be impacted a lot, as their systems remain vulnerable for a long time needed to rebuild a lot of packages, a similar problem affects binary distributions. After all, the distributions need to rebuild all affected programs in order to fully ship the fix to their end users which also involves some delay.

Comparatively, shipping a new version of a shared library takes much less time and fixes all affected programs almost instantly (modulo the necessity of restarting them).

The extreme case of static linking is to distribute proprietary software that is statically linked to its dependencies. This is primarily done to ensure that the software can be run easily on a variety of systems without requiring the user to install its dependencies manually. However, this scenario is really a form of bundling dependencies, so it will be discussed in the respective section.

However, static linking has also been historically used for system programs that were meant to keep working even if their dependent libraries became broken.

In modern packages, static linking is used for another reason entirely — because they do not require the modern programming languages to have a stable ABI. The Go compiler does not need to be concerned about emitting code that would be binary compatible with the code coming from a previous version. It works around the problem by requiring you to rebuild everything every time the compiler is upgraded.

To follow the best practices, we strongly discourage static linking in C and its derivatives. However, we can’t do much about languages such as Go or Rust that put static linking at the core of their design and have time and again stated publicly that they will not switch to dynamic linking of dependencies.

Pinning dependencies

While static linking is bad, at least it provides a reasonably clear way for automatic updates (and therefore the propagation of vulnerability fixes) to happen, pinning dependencies means requiring a specific version of your program’s dependencies to be installed. While the exact results depend on the ecosystem and the exact way of pinning the dependency, generally it means that at least some users of your package will not be able to automatically update the dependencies to newer versions.

That might not seem that bad at first. However, it means that if a bug fix or — even more importantly — a vulnerability fix is released for the dependency, the users will not get it unless you update the pin and make a new release. And then, if somebody else pins your package, then that pin will also need to be updated and released. And the chain goes on. Not to mention what happens if some package just happens to indirectly pin to two different versions of the same dependency!

Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. However, all that has another underlying problem — the combination of not being concerned with API stability on upstream part, and not wishing to unnecessarily update working code (that uses deprecated API) on downstream part. Truth is, pinning makes this worse because it sweeps the problem under the carpet, and actively encourages people to develop their code against specific versions of their dependencies rather than against a stable public API. Hyrum’s Law in practice.

Dependency pinning can have really extreme consequences. Unless you make sure to update your pins often, you may one day find yourself having to take a sudden leap — because you have relied on a very old version of a dependency that is now known to be vulnerable, and in order to update it you suddenly have to rewrite a lot of code to follow the API changes. Long term, this approach simply does not scale anymore, the effort needed to keep things working grows exponentially.

We try hard to unpin the dependencies and test packages with the newest versions of them. However, often we end up discovering that the newer versions of dependencies simply are not compatible with the packages in question. Sadly, upstreams often either ignore reports of these incompatibilities or even are actively hostile to us for not following their pins.

Bundling/vendoring dependencies

Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used.

In open source software, bundling usually means either including the sources of your dependencies along with your program or making the build system fetch them automatically, and then building them along with the program. In closed source software, it usually means linking the program to its dependencies statically or including the dependency libraries along with the program.

The baseline problem is the same as with pinned dependencies — if one of them turns out to be buggy or vulnerable, the users need to wait for a new release to update the bundled dependency. In open source software or closed source software using dynamic libraries, the packager has at least a reasonable chance of replacing the problematic dependency or unbundling it entirely (i.e. forcing the system library). In statically linked closed source software, it is often impossible to even reliably determine what libraries were actually used, not to mention their exact versions. Your distribution can no longer reliably monitor security vulnerabilities; the trust is shifted to software vendors.

However, modern software sometimes takes a step further — and vendor modified dependencies. The horror of it! Now not only the packager needs to work to replace the library but often has to actually figure out what was changed compared to the original version, and rebase the changes. In worst cases, the code becomes disconnected from upstream to the point that the program author is no longer capable of updating the vendored dependency properly.

Sadly, this kind of vendoring is becoming more common with the rapid development happening these days. The cause is twofold. On one hand, downstream consumers find it easier to fork and patch a dependency than to work with upstreams. On the other hand, many upstreams are not really concerned with fixing bugs and feature requests that do not affect their own projects. Even if the fork is considered only as a stop-gap measure, it often takes a real lot of effort to push the changes upstream afterwards and re-synchronize the codebases.

We are strongly opposed to bundling dependencies. Whenever possible, we try to unbundle them — sometimes having to actually patch the build systems to reuse system libraries. However, this is a lot of work, and often it is not even possible because of custom patching, including the kind of patching that has been explicitly rejected upstream. To list a few examples — Mozilla products rely on SQLite 3 patches that collide with regular usage of this library, Rust bundles a fork of LLVM.

Summary

Static linking, dependency pinning and bundling are three bad practices that have serious impact on the time and effort needed to eliminate vulnerabilities from production systems. They can make the difference between being able to replace a vulnerable library within a few minutes and having to spend a lot of effort and time in locating multiple copies of the vulnerable library, patching and rebuilding all the software including them.

The major Linux distributions had policies against these practices for a very long time, and have been putting a lot of effort into eliminating them. Nevertheless, it feels more and more like Sisyphean task. While we have been able to successfully resolve these problems in many packages, whole new ecosystems were built on top of these bad practices — and it does not seem that upstreams care about fixing them at all.

New programming languages such as Go and Rust rely entirely on static linking, and there’s nothing we can do about it. Instead of packaging the dependencies and having programs use the newest versions, we just fetch the versions pinned by upstream and make big blobs out of it. And while upstreams brag how they magically resolved all security issues you could ever think of (entirely ignoring other classes of security issues than memory-related), we just hope that we won’t suddenly be caught with our pants down when a common pinned dependency of many packages turns out to be vulnerable.

Added 2021-04-27: Deadlock vulnerability through embedded app-emulation/containers-storage (CVE-2021-20291) is an example how a vulnerability in a single Go package requires hunting all affected ebuilds.

February 12 2021

In Memory of Kent “kentnl” Fredric

Gentoo News (GentooNews) February 12, 2021, 6:00

Gentoo mourns the sudden loss of Kent Fredric, also known to us by his IRC handle kent\n. He passed away following a tragic accident a few days ago.

Kent was an active member of the Gentoo community for many years. He tirelessly managed Gentoo’s Perl support, and was active in the Rust project as well as in many other corners. We all remember him as an enthusiastic, bright person, with lots of eye for detail and constant willingness to help out and improve things. On behalf of the world-wide Gentoo community, our heartfelt condolences go out to his family and friends.

Please join us in remembering Kent on the Gentoo forums.

Gentoo mourns the sudden loss of Kent Fredric, also known to us by his IRC handle kent\n. He passed away following a tragic accident a few days ago.

Kent was an active member of the Gentoo community for many years. He tirelessly managed Gentoo’s Perl support, and was active in the Rust project as well as in many other corners. We all remember him as an enthusiastic, bright person, with lots of eye for detail and constant willingness to help out and improve things. On behalf of the world-wide Gentoo community, our heartfelt condolences go out to his family and friends.

Please join us in remembering Kent on the Gentoo forums.

February 10 2021

lzip decompression support for xz-utils

Michał Górny (mgorny) February 10, 2021, 14:29

As of today, the most common implementation of the LZMA algorithm on open source operating systems is the xz format. However, there are a few others available. Notably, a few packages found in the Gentoo repository use the superior lzip format. Does this mean you may end up having to have separate decompressors for both formats installed? Not necessarily.

Back in 2017, I’ve entertained a curious idea. Since both lzip and xz are both container formats built on top of the original LZMA algorithm, and xz features backwards-compatible support for the earlier container format used by lzma-utils, how hard would it be to implement a decoder for the lzip format as well? Not very hard, it turned out. After all, most of the code was already there — I’ve just had to implement the additional container format. With some kind help of XZ upstream, I’ve done that.

Sadly, the patches have not been merged yet. More than three years have passed now in waiting. Today I’ve rebased them and updated to follow changes in XZ itself. For anyone interested, it can be found on the lzip2 branch of my xz fork. After building xz with my patches, it now happily decompresses .lz files in addition to the regular set. Thanks to a tiny patchset you don’t have to build yet another program to unpack a few distfiles.

As of today, the most common implementation of the LZMA algorithm on open source operating systems is the xz format. However, there are a few others available. Notably, a few packages found in the Gentoo repository use the superior lzip format. Does this mean you may end up having to have separate decompressors for both formats installed? Not necessarily.

Back in 2017, I’ve entertained a curious idea. Since both lzip and xz are both container formats built on top of the original LZMA algorithm, and xz features backwards-compatible support for the earlier container format used by lzma-utils, how hard would it be to implement a decoder for the lzip format as well? Not very hard, it turned out. After all, most of the code was already there — I’ve just had to implement the additional container format. With some kind help of XZ upstream, I’ve done that.

Sadly, the patches have not been merged yet. More than three years have passed now in waiting. Today I’ve rebased them and updated to follow changes in XZ itself. For anyone interested, it can be found on the lzip2 branch of my xz fork. After building xz with my patches, it now happily decompresses .lz files in addition to the regular set. Thanks to a tiny patchset you don’t have to build yet another program to unpack a few distfiles.

January 31 2021

New Gentoo riscv (and arm) stages

Andreas K. Hüttel (dilfridge) January 31, 2021, 21:31

♦With the help of our infrastructure team, I've finally managed to integrate the riscv stage builds with our signing and mirroring system. So now we have a riscv tab on the installation media download page, and the mirrors carry weekly signed stage3 archives for riscv64-lp64d and riscv64-lp64, in both openrc and systemd variants.

Using the same build infrastructure based on qemu, there are now also slowly updated stages for all arm variants coming to the mirrors. Please test them, and if anything does not work as expected, file bugs! The qemu-based builds are here a temporary measure; Matt Turner (mattst88) is preparing a fast multi-core arm64 machine, where this task will move to soon.

With the help of our infrastructure team, I've finally managed to integrate the riscv stage builds with our signing and mirroring system. So now we have a riscv tab on the installation media download page, and the mirrors carry weekly signed stage3 archives for riscv64-lp64d and riscv64-lp64, in both openrc and systemd variants.

Using the same build infrastructure based on qemu, there are now also slowly updated stages for all arm variants coming to the mirrors. Please test them, and if anything does not work as expected, file bugs! The qemu-based builds are here a temporary measure; Matt Turner (mattst88) is preparing a fast multi-core arm64 machine, where this task will move to soon.

January 16 2021

Distribution Kernels: module rebuilds, better ZFS support and UEFI executables

Michał Górny (mgorny) January 16, 2021, 10:12

The primary goal of the Distribution Kernel project is provide a seamless kernel upgrade experience to Gentoo users. Initially, this meant configuring, building and installing the kernel during the @world upgrade. However, you had to manually rebuild the installed kernel modules (and @module-rebuild is still broken), and sometimes additionally rebuild the initramfs after doing that.

To address this, we have introduced a new dist-kernel USE flag. This flag is automatically added to all ebuilds installing kernel modules. When it is enabled, the linux-mod eclass adds a dependency on virtual/dist-kernel package. This virtual, in turn, is bound to the newest version of dist-kernel installed. As a result, whenever you upgrade your dist-kernel all the module packages will also be rebuilt via slot rebuilds. The manual @module-rebuild should no longer be necessary!

ZFS users have pointed out that after rebuilding sys-fs/zfs-kmod package, they need to rebuild the initramfs for Dracut to include the new module. We have combined the dist-kernel rebuild feature with pkg_postinst() to rebuild the initramfs whenever zfs-kmod is being rebuilt (and the dist-kernel is used). As a result, ZFS should no longer require any manual attention — as long as rebuilds succeed, the new kernel and initramfs should be capable of running on ZFS root once the @world upgrade finishes.

Finally, we have been asked to provide support for uefi=yes Dracut option. When this option is enabled, Dracut combines the EFI stub, kernel and generated initramfs into a single UEFI executable that can be booted directly. The dist-kernels now detect this scenario, and install the generated executable in place of the kernel, so everything works as expected. Note that due to implementation limitations, we also install an empty initramfs as otherwise kernel-install.d scripts would insist on creating another initramfs. Also note that until Dracut is fixed to use correct EFI stub path, you have to set the path manually in /etc/dracut.conf:

uefi_stub=/usr/lib/systemd/boot/efi/linuxx64.efi.stub

The primary goal of the Distribution Kernel project is provide a seamless kernel upgrade experience to Gentoo users. Initially, this meant configuring, building and installing the kernel during the @world upgrade. However, you had to manually rebuild the installed kernel modules (and @module-rebuild is still broken), and sometimes additionally rebuild the initramfs after doing that.

To address this, we have introduced a new dist-kernel USE flag. This flag is automatically added to all ebuilds installing kernel modules. When it is enabled, the linux-mod eclass adds a dependency on virtual/dist-kernel package. This virtual, in turn, is bound to the newest version of dist-kernel installed. As a result, whenever you upgrade your dist-kernel all the module packages will also be rebuilt via slot rebuilds. The manual @module-rebuild should no longer be necessary!

ZFS users have pointed out that after rebuilding sys-fs/zfs-kmod package, they need to rebuild the initramfs for Dracut to include the new module. We have combined the dist-kernel rebuild feature with pkg_postinst() to rebuild the initramfs whenever zfs-kmod is being rebuilt (and the dist-kernel is used). As a result, ZFS should no longer require any manual attention — as long as rebuilds succeed, the new kernel and initramfs should be capable of running on ZFS root once the @world upgrade finishes.

Finally, we have been asked to provide support for uefi=yes Dracut option. When this option is enabled, Dracut combines the EFI stub, kernel and generated initramfs into a single UEFI executable that can be booted directly. The dist-kernels now detect this scenario, and install the generated executable in place of the kernel, so everything works as expected. Note that due to implementation limitations, we also install an empty initramfs as otherwise kernel-install.d scripts would insist on creating another initramfs. Also note that until Dracut is fixed to use correct EFI stub path, you have to set the path manually in /etc/dracut.conf:

uefi_stub=/usr/lib/systemd/boot/efi/linuxx64.efi.stub

January 15 2021

2020 in retrospect & happy new year 2021!

Gentoo News (GentooNews) January 15, 2021, 6:00

♦ Happy New Year 2021! Due to the COVID pandemic, 2020 was a year unlike any other, and this has also impacted many open source projects. Nevertheless, at Gentoo we have made some great strides forward. While we now start into 2021 with fresh energy (and maybe soon antibodies), let’s also take a look back. We’re happy to share with our community the most exciting news of the past 12 months – including numbers on Gentoo activity, our new developers, and featured changes and improvements!

Gentoo in numbers

2020 has featured a major increase in commits to the ::gentoo repository, and especially commits from non-developers. The overall number of commits has grown from 73400 to 104500 (by 42%), while the number of commits made by non-developers has grown from 5700 (8% of total) to 11000 (10.5% of total). The latter group has featured 333 unique authors in 2019, and 391 in 2020.

The ::guru repository has thrived in 2020. While 2019 left it with merely 7 contributors and a total of 86 commits, 2020 has featured 55 different contributors and 2725 commits. GURU is a user-curated repository with a trusted user model. Come join us!

There was also a major increase in Bugzilla activity. 2020 featured almost 25500 bugs reported, compared to 15000 in 2019. This is probably largely thanks to Agostino Sarubbo’s new tinderboxing effort. The total number of bugs closed in 2020 was 23500, compared to 15000 in 2019.

New developers

We’ve finished 2020 with three significant additions to the Gentoo family (in chronological order):

  1. Max Magorsch (arzano)

    ♦ Max joined us in February to help out with Gentoo Infrastructure. Since then, he already did tons of work. Just to list a few things, he has redesigned and modernized the Gentoo websites and rewritten packages.gentoo.org into the super cool form we have today.

  2. Sam James (sam)

    ♦ Sam joined us in July, and has contributed to a lot of different projects since. He is known as an active member of the Security team and multiple arch teams, as well as someone who fixes lots of bugs in different packages.

  3. Stephan Hartmann (sultan)

    ♦ Stephan joined us in September, and immediately started working on our Chromium-related packages. He has pushed commits to upstream Chromium; hopefully he’ll deal with all the specific problems that come up in Gentoo here. Thanks to him we also have finally caught up with Windows, offering our users a packaged version of Microsoft Edge.

Featured changes

The following major changes and improvements have happened in 2020:

Packages
  • ♦ Distribution Kernels: Gentoo now supports building and installing kernels entirely via the package manager. The new kernel packages also come with an (optional) stock configuration based on well-tested Fedora kernels, to ease the entry barrier and maintenance effort of Gentoo systems.

  • ♦ Wayland: Wayland support in Gentoo has progressed greatly, making it possible to run an Xorg-free desktop. Wayland is supported with large desktop environments such as KDE Plasma and GNOME, as well as with lightweight alternatives such as Sway and Wayfire. The latter makes it also possible to use Wayland to a large extent without resorting to XWayland.

  • ♦ Lua: A new framework has been created that permits multiple versions of Lua to be installed side-by-side. The vast majority of ~arch packages have already been migrated to this framework. This way, we have finally been able to unmask new (slotted!) Lua versions.

  • ♦ Python: We have managed to almost withdraw Python 2.7 from Gentoo, and upgrade the default to Python 3.8. Python 2.7 is still available as a build-time dependency for a few packages. We have additionally patched all the vulnerabilities known from later versions of Python.

Architectures
  • ARM64: ARM64 (AArch64) support has been elevated to stable status and is no longer experimental. The ARM64 project now provides automatically generated stage3 files, and is usually one of the fastest arch teams to test packages. We have worked to bring more packages to ARM64 and make it more feasible to run a full desktop!

  • PPC64: KDE Plasma is now available on PPC64, thanks to extensive testing and keywording efforts by Georgy Yakovlev.

  • RISC-V: Work on RISC-V support has started, with particular focus on the riscv64 architecture. The RISC-V project provides stage3 files and stable profiles for the soft-float (rv64imac/lp64) and hard-float (rv64gc/lp64d) ABIs, in both systemd and OpenRC variants. The arch team has managed to run Xorg already!

  • Prefix: Gentoo Prefix is once again capable of bootstrapping on the latest macOS releases, and work is underway to modernise prefix-specific ebuilds and merge them back into the main tree - this way ensuring that users get the latest software and that maintenance burden is reduced.

  • ♦ Android: The Gentoo Android project has released a new 64bit Android prefix tarball, featuring gcc-10.1.0, binutils-2.34 and glibc-2.31 in your pocket!

Infrastructure
  • ♦ packages.gentoo.org: The packages website has received many improvements towards being a central source of information on Gentoo packages. It now shows the results of QA checks, bugs, pull requests referencing a package, and a maintainer dashboard indicating stabilization candidates and outdated versions (according to Repology). Additionally, the display can be configured for your personal preferences!

  • ♦ Bugzilla: The Infrastructure team has implemented a major improvement to Gentoo Bugzilla performance. The database has been migrated to a newer database cluster, and the backend has been switched to mod_perl.

  • CI / Tinderbox: A second active tinderboxing (build testing) effort has been started, resulting in more bugs being detected and fixed early. This also includes running a variety of QA checks, as well as minimal environment builds that are helpful in detecting missing dependencies.

Other news
  • ♦ HPC adoption: The Prefix Project has published a conference proceeding on a case study of Gentoo in high energy physics. Gentoo also sees wider adoption in the HPC community such as Compute Canada and EESSI.
Discontinued projects

While Gentoo would like to support as much as our users wish for, we could not manage to continue all of the projects we’ve started in the past. With limited resources, we had to divert our time and effort from projects showing little promise and activity. The most important projects discontinued in 2020 were:

  • Architectures: Alpha and IA64 keywords were reduced to ~arch (i.e. unstable/testing only). HPPA stable keywords were limited to the most important packages only. SH (SuperH) was removed entirely. With very small number of users of these architectures, our arch teams decided that the effort in maintaining them is too great. In case of SuperH, our last available hardware died.

  • LibreSSL: By the end of 2020, we have decided to discontinue support for LibreSSL. With little to no support from various upstream projects, the effort necessary to maintain package compatibility exceeded the gain, especially given that OpenSSL has made a lot of progress since the forking point.

Thank you!

We can here describe only a few major items, and these cover by far not all that is going on. We would like to thank all Gentoo developers for their relentless everyday Gentoo work. While they are often not recognized for this work, Gentoo could not exist without them. Cheers, and let’s make 2021 even more productive!

Gentoo Fireworks Happy New Year 2021! Due to the COVID pandemic, 2020 was a year unlike any other, and this has also impacted many open source projects. Nevertheless, at Gentoo we have made some great strides forward. While we now start into 2021 with fresh energy (and maybe soon antibodies), let’s also take a look back. We’re happy to share with our community the most exciting news of the past 12 months – including numbers on Gentoo activity, our new developers, and featured changes and improvements!

Gentoo in numbers

2020 has featured a major increase in commits to the ::gentoo repository, and especially commits from non-developers. The overall number of commits has grown from 73400 to 104500 (by 42%), while the number of commits made by non-developers has grown from 5700 (8% of total) to 11000 (10.5% of total). The latter group has featured 333 unique authors in 2019, and 391 in 2020.

The ::guru repository has thrived in 2020. While 2019 left it with merely 7 contributors and a total of 86 commits, 2020 has featured 55 different contributors and 2725 commits. GURU is a user-curated repository with a trusted user model. Come join us!

There was also a major increase in Bugzilla activity. 2020 featured almost 25500 bugs reported, compared to 15000 in 2019. This is probably largely thanks to Agostino Sarubbo’s new tinderboxing effort. The total number of bugs closed in 2020 was 23500, compared to 15000 in 2019.

New developers

We’ve finished 2020 with three significant additions to the Gentoo family (in chronological order):

  1. Max Magorsch (arzano)

    Max joined us in February to help out with Gentoo Infrastructure. Since then, he already did tons of work. Just to list a few things, he has redesigned and modernized the Gentoo websites and rewritten packages.gentoo.org into the super cool form we have today.

  2. Sam James (sam)

    Sam joined us in July, and has contributed to a lot of different projects since. He is known as an active member of the Security team and multiple arch teams, as well as someone who fixes lots of bugs in different packages.

  3. Stephan Hartmann (sultan)

    Stephan joined us in September, and immediately started working on our Chromium-related packages. He has pushed commits to upstream Chromium; hopefully he’ll deal with all the specific problems that come up in Gentoo here. Thanks to him we also have finally caught up with Windows, offering our users a packaged version of Microsoft Edge.

The following major changes and improvements have happened in 2020:

Packages

  • Distribution Kernels: Gentoo now supports building and installing kernels entirely via the package manager. The new kernel packages also come with an (optional) stock configuration based on well-tested Fedora kernels, to ease the entry barrier and maintenance effort of Gentoo systems.

  • Wayland: Wayland support in Gentoo has progressed greatly, making it possible to run an Xorg-free desktop. Wayland is supported with large desktop environments such as KDE Plasma and GNOME, as well as with lightweight alternatives such as Sway and Wayfire. The latter makes it also possible to use Wayland to a large extent without resorting to XWayland.

  • Lua: A new framework has been created that permits multiple versions of Lua to be installed side-by-side. The vast majority of ~arch packages have already been migrated to this framework. This way, we have finally been able to unmask new (slotted!) Lua versions.

  • Python: We have managed to almost withdraw Python 2.7 from Gentoo, and upgrade the default to Python 3.8. Python 2.7 is still available as a build-time dependency for a few packages. We have additionally patched all the vulnerabilities known from later versions of Python.

Architectures

  • ARM64: ARM64 (AArch64) support has been elevated to stable status and is no longer experimental. The ARM64 project now provides automatically generated stage3 files, and is usually one of the fastest arch teams to test packages. We have worked to bring more packages to ARM64 and make it more feasible to run a full desktop!

  • PPC64: KDE Plasma is now available on PPC64, thanks to extensive testing and keywording efforts by Georgy Yakovlev.

  • RISC-V: Work on RISC-V support has started, with particular focus on the riscv64 architecture. The RISC-V project provides stage3 files and stable profiles for the soft-float (rv64imac/lp64) and hard-float (rv64gc/lp64d) ABIs, in both systemd and OpenRC variants. The arch team has managed to run Xorg already!

  • Prefix: Gentoo Prefix is once again capable of bootstrapping on the latest macOS releases, and work is underway to modernise prefix-specific ebuilds and merge them back into the main tree - this way ensuring that users get the latest software and that maintenance burden is reduced.

  • Android: The Gentoo Android project has released a new 64bit Android prefix tarball, featuring gcc-10.1.0, binutils-2.34 and glibc-2.31 in your pocket!

Infrastructure

  • packages.gentoo.org: The packages website has received many improvements towards being a central source of information on Gentoo packages. It now shows the results of QA checks, bugs, pull requests referencing a package, and a maintainer dashboard indicating stabilization candidates and outdated versions (according to Repology). Additionally, the display can be configured for your personal preferences!

  • Bugzilla: The Infrastructure team has implemented a major improvement to Gentoo Bugzilla performance. The database has been migrated to a newer database cluster, and the backend has been switched to mod_perl.

  • CI / Tinderbox: A second active tinderboxing (build testing) effort has been started, resulting in more bugs being detected and fixed early. This also includes running a variety of QA checks, as well as minimal environment builds that are helpful in detecting missing dependencies.

Other news

Discontinued projects

While Gentoo would like to support as much as our users wish for, we could not manage to continue all of the projects we’ve started in the past. With limited resources, we had to divert our time and effort from projects showing little promise and activity. The most important projects discontinued in 2020 were:

  • Architectures: Alpha and IA64 keywords were reduced to ~arch (i.e. unstable/testing only). HPPA stable keywords were limited to the most important packages only. SH (SuperH) was removed entirely. With very small number of users of these architectures, our arch teams decided that the effort in maintaining them is too great. In case of SuperH, our last available hardware died.

  • LibreSSL: By the end of 2020, we have decided to discontinue support for LibreSSL. With little to no support from various upstream projects, the effort necessary to maintain package compatibility exceeded the gain, especially given that OpenSSL has made a lot of progress since the forking point.

Thank you!

We can here describe only a few major items, and these cover by far not all that is going on. We would like to thank all Gentoo developers for their relentless everyday Gentoo work. While they are often not recognized for this work, Gentoo could not exist without them. Cheers, and let’s make 2021 even more productive!

December 29 2020

OpenSSL, LibreSSL, LibreTLS and all the terminological irony

Michał Górny (mgorny) December 29, 2020, 18:11

While we’re discussing the fate of LibreSSL, it’s worth noting how confusing the names of these packages became. I’d like to take this opportunity to provide a short note on what’s what.

First of all, SSL and its successor TLS are protocols used to implement network connection security. For historical reasons, many libraries carry ‘SSL’ in their name (OpenSSL, LibreSSL, PolarSSL) but nowadays they all support TLS.

OpenSSL is the ‘original’ crypto/SSL/TLS library. It is maintained independently of a specific operating system. It provides two main libraries: libcrypto and libssl (that also implements TLS).

LibreSSL is a fork of OpenSSL. It is maintained by OpenBSD as part of its base system. However, the upstream also maintains LibreSSL-portable repository that provides build system and portability glue for using it on other systems. LibreSSL provides partially compatible versions of libcrypto and libssl, and a new libtls library. Both libssl and libtls can be used for TLS support in your applications.

LibreTLS is a lightweight fork of libtls from LibreSSL that builds it against OpenSSL. This makes it possible to build programs written for libtls against OpenSSL+LibreTLS instead of LibreSSL.

So, to summarize. OpenSSL is the original, while LibreSSL is the OpenBSD fork. libtls is the LibreSSL original library, while LibreTLS is its fork for OpenSSL. Makes sense, right? And finally, despite the name, they all implement TLS.

While we’re discussing the fate of LibreSSL, it’s worth noting how confusing the names of these packages became. I’d like to take this opportunity to provide a short note on what’s what.

First of all, SSL and its successor TLS are protocols used to implement network connection security. For historical reasons, many libraries carry ‘SSL’ in their name (OpenSSL, LibreSSL, PolarSSL) but nowadays they all support TLS.

OpenSSL is the ‘original’ crypto/SSL/TLS library. It is maintained independently of a specific operating system. It provides two main libraries: libcrypto and libssl (that also implements TLS).

LibreSSL is a fork of OpenSSL. It is maintained by OpenBSD as part of its base system. However, the upstream also maintains LibreSSL-portable repository that provides build system and portability glue for using it on other systems. LibreSSL provides partially compatible versions of libcrypto and libssl, and a new libtls library. Both libssl and libtls can be used for TLS support in your applications.

LibreTLS is a lightweight fork of libtls from LibreSSL that builds it against OpenSSL. This makes it possible to build programs written for libtls against OpenSSL+LibreTLS instead of LibreSSL.

So, to summarize. OpenSSL is the original, while LibreSSL is the OpenBSD fork. libtls is the LibreSSL original library, while LibreTLS is its fork for OpenSSL. Makes sense, right? And finally, despite the name, they all implement TLS.

November 06 2020

Renaming and reshaping Scylla tables using scylla-migrator

Alexys Jacob (ultrabug) November 06, 2020, 20:11

We have recently faced a problem where some of the first Scylla tables we created on our main production cluster were not in line any more with the evolved schemas that recent tables are using.

This typical engineering problem requires either to keep those legacy tables and data queries or to migrate it to the more optimal model with the bandwagon of applications to be modified to query the data the new way… That’s something nobody likes doing but hey, we don’t like legacy at Numberly so let’s kill that one!

To overcome this challenge we used the scylla-migrator project and I thought it could be useful to share this experience.

How and why our schema evolved

When we first approached ID matching tables we chose to answer two problems at the same time: query the most recent data and keep the history of the changes per source ID.

This means that those tables included a date as part of their PRIMARY KEY while the partition key was obviously the matching table ID we wanted to lookup from:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
date timestamp,
PRIMARY KEY ((partnerid), date, id)
)
WITH CLUSTERING ORDER BY (date DESC)

Making a table with an ever changing date in the clustering key creates what we call a history table. In the schema above the uniqueness of a row is not only defined by a partner_id / id couple but also by its date!

Quick caveat: you have to be careful about the actual date timestamp resolution since you may not want to create a row for every second of the same partner_id / id couple (we use an hour resolution).

History tables are good for analytics and we also figured we could use them for batch and real time queries where we would be interested in the “most recent ids for the given partner_id” (sometimes flavored with a LIMIT):

SELECT id FROM ids_by_partnerid WHERE partner_id = "AXZAZLKDJ" ORDER BY date DESC;

As time passed, real time Kafka pipelines started to query these tables hard and were mostly interested in “all the ids known for the given partner_id“.

A sort of DISTINCT(id) is out of the scope of our table! For this we need a table schema that represents a condensed view of the data. We call them compact tables and the only difference with the history table is that the date timestamp is simply not part of the PRIMARY KEY:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
seen_date timestamp,
PRIMARY KEY ((partnerid), id)
)

To make that transition happen we thus wanted to:

  • rename history tables with an _history suffix so that they are clearly identified as such
  • get a compacted version of the tables (by keeping their old name) while renaming the date column name to seen_date
  • do it as fast as possible since we will need to stop our feeding pipeline and most of our applications during the process…

STOP: it’s not possible to rename a table in CQL!

♦ Scylla-migrator to the rescue

We decided to abuse the scylla-migrator to perform this perilous migration.

As it was originally designed to help users migrate from Cassandra to Scylla by leveraging Spark it seemed like a good fit for the task since we happen to own Spark clusters powered by Hadoop YARN.

Building scylla-migrator for Spark < 2.4

Recent scylla-migrator does not support older Spark versions. The trick is to look at the README.md git log and checkout the hopefully right commit that supports your Spark cluster version.

In our case for Spark 2.3 we used git commit bc82a57e4134452f19a11cd127bd4c6a25f75020.

On Gentoo, make sure to use dev-java/sbt-bin since the non binary version is vastly out of date and won’t build the project. You need at least version 1.3.

The scylla-migrator plan

The documentation explains that we need a config file that points to a source cluster+table and a destination cluster+table as long as they have the same schema structure…

Renaming is then as simple as duplicating the schema using CQLSH and running the migrator!

But what about our compacted version of our original table? The schema is different from the source table!…

Good news is that as long as all your columns remain present, you can also change the PRIMARY KEY of your destination table and it will still work!

This make the scylla-migrator an amazing tool to reshape or pivot tables!

  • the column date is renamed to seen_date: that’s okay, scylla-migrator supports column renaming (it’s a Spark dataframe after all)!
  • the PRIMARY KEY is different in the compacted table since we removed the ‘date‘ from the clustering columns: we’ll get a compacted table for free!
Using scylla-migrator

The documentation is a bit poor on how to submit your application to a Hadoop YARN cluster but that’s kind of expected.

It also did not mention how to connect to a SSL enabled cluster (are there people really not using SSL on the wire in their production environment?)… anyway let’s not start a flame war ♦

The trick that will save you is to know that you can append all the usual Spark options that are available in the spark-cassandra-connector!

Submitting to a Kerberos protected Hadoop YARN cluster targeting a SSL enabled Scylla cluster then looks like this:

export JAR_NAME=target/scala-2.11/scylla-migrator-assembly-0.0.1.jar
export KRB_PRINCIPAL=USERNAME

spark2-submit \
 --name ScyllaMigratorApplication \
 --class com.scylladb.migrator.Migrator  \
 --conf spark.cassandra.connection.ssl.clientAuth.enabled=True  \
 --conf spark.cassandra.connection.ssl.enabled=True  \
 --conf spark.cassandra.connection.ssl.trustStore.path=jssecacerts  \
 --conf spark.cassandra.connection.ssl.trustStore.password=JKS_PASSWORD  \
 --conf spark.cassandra.input.consistency.level=LOCAL_QUORUM \
 --conf spark.cassandra.output.consistency.level=LOCAL_QUORUM \
 --conf spark.scylla.config=config.yaml \
 --conf spark.yarn.executor.memoryOverhead=1g \
 --conf spark.blacklist.enabled=true  \
 --conf spark.blacklist.task.maxTaskAttemptsPerExecutor=1  \
 --conf spark.blacklist.task.maxTaskAttemptsPerNode=1  \
 --conf spark.blacklist.stage.maxFailedTasksPerExecutor=1  \
 --conf spark.blacklist.stage.maxFailedExecutorsPerNode=1  \
 --conf spark.executor.cores=16 \
 --deploy-mode client \
 --files jssecacerts \
 --jars ${JAR_NAME}  \
 --keytab ${KRB_PRINCIPAL}.keytab  \
 --master yarn \
 --principal ${KRB_PRINCIPAL}  \
 ${JAR_NAME}

Note that we chose to apply a higher consistency level to our reads using a LOCAL_QUORUM instead of the default LOCAL_ONE. I strongly encourage you to do the same since it’s appropriate when you’re using this kind of tool!

Column renaming is simply expressed in the configuration file like this:

# Column renaming configuration.
renames:
  - from: date
    to: seen_date
Tuning scylla-migrator

While easy to use, tuning scylla-migrator to operate those migrations as fast as possible turned out to be a real challenge (remember we have some production applications shut down during the process).

Even using 300+ Spark executors I couldn’t get my Scylla cluster utilization to more than 50% and migrating a single table with a bit more than 1B rows took almost 2 hours…

We found the best knobs to play with thanks to the help of Lubos Kosco and this blog post from ScyllaDB:

  • Increase the splitCount setting: more splits means more Spark executors will be spawned and more tasks out of it. While it might be magic on a pure Spark deployment it’s not that amazing on a Hadoop YARN one where executors are scheduled in containers with 1 core by default. We simply moved it from 256 to 384.
  • Disable compaction on destination tables schemas. This gave us a big boost and saved the day since it avoids adding the overhead of compacting while you’re pushing down data hard!

To disable compaction on a table simply:

ALTER TABLE ids_by_partnerid_history WITH compaction = {'class': 'NullCompactionStrategy'};

Remember to run a manual compaction (nodetool compact <keyspace> <table>) and to enable compaction back on your tables once you’re done!

Happy Scylla tables mangling!

We have recently faced a problem where some of the first Scylla tables we created on our main production cluster were not in line any more with the evolved schemas that recent tables are using.

This typical engineering problem requires either to keep those legacy tables and data queries or to migrate it to the more optimal model with the bandwagon of applications to be modified to query the data the new way… That’s something nobody likes doing but hey, we don’t like legacy at Numberly so let’s kill that one!

To overcome this challenge we used the scylla-migrator project and I thought it could be useful to share this experience.

How and why our schema evolved

When we first approached ID matching tables we chose to answer two problems at the same time: query the most recent data and keep the history of the changes per source ID.

This means that those tables included a date as part of their PRIMARY KEY while the partition key was obviously the matching table ID we wanted to lookup from:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
date timestamp,
PRIMARY KEY ((partnerid), date, id)
)
WITH CLUSTERING ORDER BY (date DESC)

Making a table with an ever changing date in the clustering key creates what we call a history table. In the schema above the uniqueness of a row is not only defined by a partner_id / id couple but also by its date!

Quick caveat: you have to be careful about the actual date timestamp resolution since you may not want to create a row for every second of the same partner_id / id couple (we use an hour resolution).

History tables are good for analytics and we also figured we could use them for batch and real time queries where we would be interested in the “most recent ids for the given partner_id” (sometimes flavored with a LIMIT):

SELECT id FROM ids_by_partnerid WHERE partner_id = "AXZAZLKDJ" ORDER BY date DESC;

As time passed, real time Kafka pipelines started to query these tables hard and were mostly interested in “all the ids known for the given partner_id“.

A sort of DISTINCT(id) is out of the scope of our table! For this we need a table schema that represents a condensed view of the data. We call them compact tables and the only difference with the history table is that the date timestamp is simply not part of the PRIMARY KEY:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
seen_date timestamp,
PRIMARY KEY ((partnerid), id)
)

To make that transition happen we thus wanted to:

  • rename history tables with an _history suffix so that they are clearly identified as such
  • get a compacted version of the tables (by keeping their old name) while renaming the date column name to seen_date
  • do it as fast as possible since we will need to stop our feeding pipeline and most of our applications during the process…

STOP: it’s not possible to rename a table in CQL!

Scylla-migrator to the rescue

We decided to abuse the scylla-migrator to perform this perilous migration.

As it was originally designed to help users migrate from Cassandra to Scylla by leveraging Spark it seemed like a good fit for the task since we happen to own Spark clusters powered by Hadoop YARN.

Building scylla-migrator for Spark < 2.4

Recent scylla-migrator does not support older Spark versions. The trick is to look at the README.md git log and checkout the hopefully right commit that supports your Spark cluster version.

In our case for Spark 2.3 we used git commit bc82a57e4134452f19a11cd127bd4c6a25f75020.

On Gentoo, make sure to use dev-java/sbt-bin since the non binary version is vastly out of date and won’t build the project. You need at least version 1.3.

The scylla-migrator plan

The documentation explains that we need a config file that points to a source cluster+table and a destination cluster+table as long as they have the same schema structure…

Renaming is then as simple as duplicating the schema using CQLSH and running the migrator!

But what about our compacted version of our original table? The schema is different from the source table!…

Good news is that as long as all your columns remain present, you can also change the PRIMARY KEY of your destination table and it will still work!

This make the scylla-migrator an amazing tool to reshape or pivot tables!

  • the column date is renamed to seen_date: that’s okay, scylla-migrator supports column renaming (it’s a Spark dataframe after all)!
  • the PRIMARY KEY is different in the compacted table since we removed the ‘date‘ from the clustering columns: we’ll get a compacted table for free!

Using scylla-migrator

The documentation is a bit poor on how to submit your application to a Hadoop YARN cluster but that’s kind of expected.

It also did not mention how to connect to a SSL enabled cluster (are there people really not using SSL on the wire in their production environment?)… anyway let’s not start a flame war 🙂

The trick that will save you is to know that you can append all the usual Spark options that are available in the spark-cassandra-connector!

Submitting to a Kerberos protected Hadoop YARN cluster targeting a SSL enabled Scylla cluster then looks like this:

export JAR_NAME=target/scala-2.11/scylla-migrator-assembly-0.0.1.jar
export KRB_PRINCIPAL=USERNAME

spark2-submit \
 --name ScyllaMigratorApplication \
 --class com.scylladb.migrator.Migrator  \
 --conf spark.cassandra.connection.ssl.clientAuth.enabled=True  \
 --conf spark.cassandra.connection.ssl.enabled=True  \
 --conf spark.cassandra.connection.ssl.trustStore.path=jssecacerts  \
 --conf spark.cassandra.connection.ssl.trustStore.password=JKS_PASSWORD  \
 --conf spark.cassandra.input.consistency.level=LOCAL_QUORUM \
 --conf spark.cassandra.output.consistency.level=LOCAL_QUORUM \
 --conf spark.scylla.config=config.yaml \
 --conf spark.yarn.executor.memoryOverhead=1g \
 --conf spark.blacklist.enabled=true  \
 --conf spark.blacklist.task.maxTaskAttemptsPerExecutor=1  \
 --conf spark.blacklist.task.maxTaskAttemptsPerNode=1  \
 --conf spark.blacklist.stage.maxFailedTasksPerExecutor=1  \
 --conf spark.blacklist.stage.maxFailedExecutorsPerNode=1  \
 --conf spark.executor.cores=16 \
 --deploy-mode client \
 --files jssecacerts \
 --jars ${JAR_NAME}  \
 --keytab ${KRB_PRINCIPAL}.keytab  \
 --master yarn \
 --principal ${KRB_PRINCIPAL}  \
 ${JAR_NAME}

Note that we chose to apply a higher consistency level to our reads using a LOCAL_QUORUM instead of the default LOCAL_ONE. I strongly encourage you to do the same since it’s appropriate when you’re using this kind of tool!

Column renaming is simply expressed in the configuration file like this:

# Column renaming configuration.
renames:
  - from: date
    to: seen_date

Tuning scylla-migrator

While easy to use, tuning scylla-migrator to operate those migrations as fast as possible turned out to be a real challenge (remember we have some production applications shut down during the process).

Even using 300+ Spark executors I couldn’t get my Scylla cluster utilization to more than 50% and migrating a single table with a bit more than 1B rows took almost 2 hours…

We found the best knobs to play with thanks to the help of Lubos Kosco and this blog post from ScyllaDB:

  • Increase the splitCount setting: more splits means more Spark executors will be spawned and more tasks out of it. While it might be magic on a pure Spark deployment it’s not that amazing on a Hadoop YARN one where executors are scheduled in containers with 1 core by default. We simply moved it from 256 to 384.
  • Disable compaction on destination tables schemas. This gave us a big boost and saved the day since it avoids adding the overhead of compacting while you’re pushing down data hard!

To disable compaction on a table simply:

ALTER TABLE ids_by_partnerid_history WITH compaction = {'class': 'NullCompactionStrategy'};

Remember to run a manual compaction (nodetool compact <keyspace> <table>) and to enable compaction back on your tables once you’re done!

Happy Scylla tables mangling!

VIEW

SCOPE

FILTER
  from
  to