May 06 2021

10 Years’ Perspective on Python in Gentoo

Michał Górny (mgorny) May 06, 2021, 7:55

I’m a Gentoo developer for over 10 years already. I’ve been doing a lot of different things throughout that period. However, Python was pretty much always somewhere within my area of interest. I don’t really recall how it all started. Maybe it had something to do with Portage being written in Python. Maybe it was the natural next step after programming in Perl.

I feel like the upcoming switch to Python 3.9 is the last step in the prolonged effort of catching up with Python. Over the last years, we’ve been working real hard to move Python support forward, to bump neglected packages, to enable testing where tests are available, to test packages on new targets and unmask new targets as soon as possible. We have improved the processes a lot. Back when we were switching to Python 3.4, it took almost a year from the first false start attempt to the actual change. We started using Python 3.5 by default after upstream dropped bugfix support for it. In a month from now, we are going to start using Python 3.9 even before 3.10 final is released.

I think this is a great opportunity to look back and see what changed in the Gentoo Python ecosystem, in the last 10 years.

Python package ebuilds 10 years ago

Do you know how a Python package ebuild looked like 10 years ago? Let’s take gentoopm-0.1 as an example (reformatted to fit the narrow layout better):


# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.1.ebuild,v 1.1 2011/07/15 19:05:26 mgorny Exp $

EAPI=3

PYTHON_DEPEND='*:2.6'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS='2.4 2.5'
DISTUTILS_SRC_TEST=setup.py

inherit base distutils

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="github.com/gentoopm/"
SRC_URI="cloud.github.com/downloads/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.8.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="dev-python/epydoc"
PDEPEND="app-admin/eselect-package-manager"

src_prepare() {
  base_src_prepare
  distutils_src_prepare
}

src_compile() {
  distutils_src_compile

  if use doc; then
    "$(PYTHON -2)" setup.py doc || die
  fi
}

src_install() {
  distutils_src_install

  if use doc; then
    dohtml -r doc/* || die
  fi
}

This ebuild is actually using the newer API of python.eclass that is enabled via SUPPORT_PYTHON_ABIS. It provides support for installing for multiple implementations (like the modern python-r1 eclass). PYTHON_DEPEND is used to control the dependency string added to ebuild. The magical syntax here means that the ebuild supports both Python 2 and Python 3, from Python 2.6 upwards. RESTRICT_PYTHON_ABIS opts out support for Python versions prior to 2.6. Note the redundancy — PYTHON_DEPEND controls the dependency, specified as a range of Python 2 and/or Python 3 versions, RESTRICT_PYTHON_ABIS controls versions used at build time and needs to explicitly exclude all unsupported branches.

Back then, there were no PYTHON_TARGETS to control what was built. Instead, the eclass defaulted to using whatever was selected via eselect python, with the option to override it via setting USE_PYTHON in make.conf. Therefore, there were no cross-package USE dependencies and you had to run python-updater to verify whether all packages are built for the current interpreter, and rebuild these that were not.

Still, support for multiple ABIs, as the eclass called different branches/implementations of Python, was a major step forward. It was added around the time that the first releases of Python 3 were published, and our users have been relying on it to support a combination of Python 2 and Python 3 for years. Today, we’re primarily using it to aid developers in testing their packages and to provide a safer upgrade experience.

The python.eclass stalemate

Unfortunately, things at the time were not all that great. Late 2010 marks a conflict between the primary Python developer and the rest of the community, primarily due to the major breakage being caused by the changes in Python support. By mid-2011, it was pretty clear that there is no chance to resolve the conflict. The in-Gentoo version of python.eclass was failing to get EAPI 4 support for 6 months already, while an incompatible version continued being developed in the (old) Python overlay. As Dirkjan Ochtman related in his mail:

I guess by now pretty much everyone knows that the python eclass is rather complex, and that this poses some problems. This has also been an important cause for the disagreements between Arfrever and some of the other developers. Since it appears that Arfrever won’t be committing much code to gentoo-x86 in the near future, I’m trying to figure out where we should go with the python.eclass. […]

Dirkjan Ochtman, 2011-06-27, [gentoo-dev] The Python problem

Eventually, some of the changes from the Python overlay were backported to the eclass and EAPI 4 support was added. Nevertheless, at this point it was pretty clear that we need a new way forward. Unfortunately, the discussions were leading nowhere. With the primary eclass maintainer retired, nobody really comprehended most of the eclass, nor were able to afford the time to figure it out. At the same time, involved parties wanted to preserve backwards compatibility while moving forward.

The tie breaker: python-distutils-ng

Some of you might find it surprising that PYTHON_TARGETS are not really a python-r1 invention. Back in March 2012, when Python team was still unable to find a way forward with python.eclass, Krzysztof Pawlik (nelchael) has committed a new python-distutils-ng.eclass. It has never grown popular, and it has been replaced by the python-r1 suite before it ever started being a meaningful replacement for python.eclass. Still, it served an important impulse that made what came after possible.

Here’s a newer gentoopm ebuild using the new eclass (again reformatted):


# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.2.5-r1.ebuild,v 1.1 2012/05/26 10:11:21 mgorny Exp $

EAPI=4
PYTHON_COMPAT='python2_6 python2_7 python3_1 python3_2'

inherit base python-distutils-ng

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="github.com/mgorny/gentoopm/"
SRC_URI="mirror://github/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~mips ~x86 ~x86-fbsd"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.10.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="doc? ( dev-python/epydoc )"
PDEPEND="app-admin/eselect-package-manager"

python_prepare_all() {
  base_src_prepare
}

src_compile() {
  python-distutils-ng_src_compile
  if use doc; then
    "${PYTHON}" setup.py doc || die
  fi
}

python_install_all() {
  if use doc; then
    dohtml -r doc/*
  fi
}

Just looking at the code, you may see that python-r1 has inherited a lot after this eclass. python-distutils-ng in turn followed some of the good practices introduced before in the ruby-ng eclass. It introduced PYTHON_TARGETS to provide explicit visible control over implementations used for the build — though notably it did not include a way for packages to depend on matching flags (i.e. the equivalent of ${PYTHON_USEDEP}). It also used the sub-phase approach that makes distutils-r1 and ruby-ng eclasses much more convenient than the traditional python.eclass approach that roughly resembled using python_foreach_impl all the time.

What’s really important is that python-distutils-ng carved a way forward. It’s been a great inspiration and a proof of concept. It has shown that we do not have to preserve compatibility with python.eclass forever, or have to learn its inner workings before starting to solve problems. I can’t say how Python would look today if it did not happen but I can say with certainly that python-r1 would not happen so soon if it weren’t for it.

python-r1

In October 2012, the first version of python-r1 was committed. It combined some of the very good ideas of python-distutils-ng with some of my own.  The goal was not to provide an immediate replacement for python.eclass. Instead, the plan was to start simple and add new features as they turned out to be necessary. Not everything went perfectly but I dare say that the design has stood the test of time. While I feel like the eclasses ended up being more complex than I wished they would be, they still work fine with no replacement in sight and they serve as inspiration to other eclasses.

For completeness, here’s a 2017 gentoopm live ebuild that uses a pretty complete distutils-r1 feature set:


# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6
PYTHON_COMPAT=( python{2_7,3_4,3_5,3_6} pypy )

EGIT_REPO_URI="github.com/mgorny/gentoopm.git"
inherit distutils-r1 git-r3

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="github.com/mgorny/gentoopm/"
SRC_URI=""

LICENSE="BSD-2"
SLOT="0"
KEYWORDS=""
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/pkgcore-0.9.4[${PYTHON_USEDEP}]
    >=sys-apps/portage-2.1.10.3[${PYTHON_USEDEP}]
    >=sys-apps/paludis-3.0.0_pre20170219[python,${PYTHON_USEDEP}]
  )"
DEPEND="
  doc? (
    dev-python/epydoc[$(python_gen_usedep python2_7)]
  )"
PDEPEND="app-eselect/eselect-package-manager"

REQUIRED_USE="
  doc? ( $(python_gen_useflags python2_7) )"

src_configure() {
  use doc && DISTUTILS_ALL_SUBPHASE_IMPLS=( python2.7 )
  distutils-r1_src_configure
}

python_compile_all() {
  use doc && esetup.py doc
}

python_test() {
  esetup.py test
}

python_install_all() {
  use doc && local HTML_DOCS=( doc/. )
  distutils-r1_python_install_all
}

The new eclasses have employed the same PYTHON_TARGETS flags for the general implementation choice but also added PYTHON_SINGLE_TARGET to make choosing the implementation more convenient (and predictable at the same time) when the package did not permit choosing more than one. The distutils-r1 eclass reused the great idea of sub-phases to make partial alterations to the phases easier.

The unique ideas included:

  • the split into more eclasses by functionality (vs switching the mode via variables as done in python.eclass)
  • exposing dependencies and REQUIRED_USE constraints via variables to be used in ebuild instead of writing elaborate mechanisms for adding dependencies
  • avoiding command substitution in global scope to keep metadata regeneration fast

The eclasses evolved a lot over the years. Some of the original ideas turned out pretty bad, e.g. trying to run sub-phases in parallel (which broke a lot of stuff for minor performance gain) or the horribly complex original interaction between PYTHON_TARGETS and PYTHON_SINGLE_TARGET. Ebuilds often missed dependencies and REQUIRED_USE constraints, until we finally made the pkgcheck-based CI report that.

The migration to new eclasses took many years. Initially, python-r1 ebuilds were even depending on python.eclass ebuilds. The old eclass was not removed until March 2017, i.e. 4.5 years after introducing the new eclasses.

Testing on new Python targets

We went for the opt-in approach with PYTHON_COMPAT. This means that for every new Python target added, we start with no packages supporting it and have to iterate over all of the packages adding the support. It’s a lot of work and it has repeatedly caused users pain due to packages not being ported in time for the big switch to the next version. Some people have complained about that and suggested that we should go for opt-out instead. However, if you think about it, opt-in is the only way to go.

The big deal is that for any particular package to support a new implementation, all of its Python dependencies need to support it as well. With the opt-in approach, it means that we’re doing the testing dependency-first, and reaching the bigger packages only when we confirm that there’s at least a single version of every dependency that works for it. If we do things right, users don’t even see any regressions.

If we went for the opt-out approach, all packages would suddenly claim to support the new version. Now, this wouldn’t be that bad if we were actually able to do a big CI run for all packages — but we can’t since a lot of them do not have proper tests, and Python version incompatibility often can’t be detected statically. In the end, we would be relying on someone (possibly an user) reporting that something is broken. Then we’d have to investigate where in the dependency chain the culprit is, and either restrict the new target (and then restrict it in all its reverse dependencies) or immediately fix it.

So in the end, opt-out would be worse for both users and developers. Users would hit package issues first hand, and developers would have to spend significant time on the back-and-forth effort of removing support for new targets, and then adding it again. If we are to add new targets early (which is a worthwhile goal), we have to expect incompatible packages. My experience so far shows that Gentoo developers sometimes end up being the first people to submit patches fixing the incompatibility. This can be a real problem given that many Python packages have slow release cycles, and are blocking their reverse dependencies, and these in turn block their reverse dependencies and so on.

Now, things did not always go smoothly. I have prepared a Python release and Gentoo packaging timeline that puts our packaging work into perspective. As you can see, we were always quite fast in packaging new Python interpreters but it took a significant time to actually switch the default targets to them — in fact, we often switched just before or even after upstream stopped providing bug fixes to the version in question.

Our approach has changed over the years. Early on, we generally kept both the interpreter and the target in ~arch, and stabilized them in order to switch targets. The prolonged stable-mask of the new target has resulted in inconsistent presence of support for the new target in stable, and this in turn involved a lot of last-minute stabilization work. Even then, we ended up switching targets before stable was really ready for that. This was particularly bad for Python 3.4 — the period seen as ‘stable’ on the timeline is actually a period following the first unsuccessful switch of the default. It took us over half a year to try again.

Then (around Python 3.6, if I’m not mistaken) we switched to a different approach. Instead of delaying till the expected switch, we’ve tried to unmask the target on stable systems as soon as possible. This way, we started enforcing dependency graph consistency earlier and were able to avoid big last minute stabilizations needed to unmask the target.

Eventually, thanks to pkgcheck’s superior StableRequestCheck, I’ve started proactively stabilizing new versions of Python packages. This was probably the most important improvement of all. The stabilization effort was streamlined, new versions of packages gained stable keywords sooner and along them did the support for new Python implementations.

The effort in package testing and stabilizations have finally made it possible to catch up with upstream. We have basically gone through a target switching sprint, moving through 3.7 and 3.8 in half a year each. The upcoming switch to Python 3.9 concludes this effort. For the first time in years, our default will be supported upstream for over 6 months.

That said, it is worth noting that things are not becoming easier for us. Over time, Python packages keep getting new dependencies, and this means that every new Python version will involve more and more porting work. Unfortunately, some high profile packages such as setuptools and pytest keep creating bigger and bigger dependency loops. At this point, it is no longer reasonable to attempt to port all the cyclic dependencies simultaneously. Instead, I tend to temporarily disable tests for the initial ports to reduce the number of dependencies. I’ve included a list of suggested initial steps in the Python Guide to ease future porting efforts.

Packaging the Python interpreter

CPython is not the worst thing to package but it’s not the easiest one either. Admittedly, the ebuilds were pretty good back when I joined. However, we’ve already carried a pretty large and mostly undocumented set of patches, and most of these patches we carry up to this day, with no chance of upstreaming them.

Some of the involved patches are build fixes and hacks that are either specific to Gentoo, or bring the flexibility Gentoo cares about (such as making some of the USE flags possible). There are also some fixes for old bugs that upstream has not shown any interest in fixing.

Another part of release management is resolving security bugs. Until recently, we did not track vulnerabilities in CPython very well. Thanks to the work of new Security team members, we have started being informed of vulnerabilities earlier. At the same time, we realized that CPython’s treatment of vulnerabilities is a bit suboptimal.

Admittedly, when very bad things happen upstream releases fixes quickly. However, non-critical vulnerability fixes are released as part of the normal release cycle, approximately every two months. For some time already, every new release contained some security fixes and had to be stabilized quickly. At the same time it contained many other changes with their own breakage potential. Old Python branches were even worse — absurdly, even though these versions received security fixes only, the releases were even rarer.

In the end, I’ve decided that it makes more sense to backport security fixes to our patchset as soon as I become aware of them, and stabilize the patch bumps instead. I do this even if upstream makes a new release at the same time, since patch bumps are safer stable targets. Even then, some of the security fixes actually require changing the behavior. To name a few recent changes:

  • urllib.parse.parse_qsl() historically used to split the URL query string on either & or ;. This somewhat surprising behavior was changed to split only on &, with a parameter to change the separator (but no option to restore the old behavior).
  • urllib.parse.urlparse() historically preserved newlines, CRs and tabs. In the latest Python versions this behavior was changed to follow a newer recommendation of stripping these characters. As a side effect, some URL validators (e.g. in Django) suddenly stopped rejecting URLs with newlines.
  • The ipaddress module recently stopped allowing leading zeros in IPv4 addresses. These were accepted before but some of the external libraries were incidentally interpreting them as octal numbers.

Some of these make you scratch your head.

Unsurprisingly, this is also a lot of work. At this very moment, we are maintaining six different slots of CPython (2.7, 3.6, 3.7, 3.8, 3.9, 3.10). For every security backport set, I start by identifying the security-related commits on the newest release branch. This is easy if they’re accompanied by a news entry in Security category — unfortunately, some vulnerability fixes were treated as regular bug fixes in the past. Once I have a list of commits, I cherry-pick them to our patched branch and make a patchset out of that. This is the easy part.

Now I have to iterate over all the older versions. For maintained branches, the first step is to identify if upstream has already backported their fixes. if they did, I cherry-pick the backport commits from the matching branch. If they did not, I need to verify if the vulnerability applies to this old version, and cherry-pick from a newer version. Sometimes it requires small changes.

The hardest part is Python 2.7 — it is not supported upstream but still used as a build-time dependency by a few projects. The standard library structure in Python 2 differs a lot from the one in Python 3 but many of Python 3 vulnerabilities stem from code dating back to Python 2. In the end, I have to find the relevant piece of original code in Python 2, see if it is vulnerable and usually rewrite the patch for the old code.

I wish this were the end. However, in Gentoo we are also packaging PyPy, the alternative Python implementation. PyPy follows its own release cycle. PyPy2 is based on now-unmaintained Python 2, so most of the vulnerability fixes from our Python 2.7 patchset need to be ported to it. PyPy3 tries to follow Python 3’s standard library but is not very fast at it. In the end, I end up cherry-picking the changes from CPython to PyPy for our patchsets, and at the same time sending them upstream’s way so that they can fix the vulnerabilities without having to sync with CPython immediately.

Historically, we have also supported another alternative implementation, Jython. However, for a very long time upstream’s been stuck on Python 2.5 and we’ve eventually dropped support for Jython as it became unmaintainable. Upstream has eventually caught up with Python 2.7 but you can imagine how useful that is today.

There are other interesting projects out there but I don’t think we have the manpower (or even a very good reason) to work on them.

Now, a fun fact: Victor Stinner indicates in his mail Need help to fix known Python security vulnerabilities that the security fixes CPython receives are only a drop in the ocean. Apparently, at the time of writing there were 78 open problems of various severity.

A bus and a phoenix

Any project where a single developer does a sufficiently large portion of work has a tendency towards a bus factor of one. The Python project was no different. Ten years ago we were relying on a single person doing most of the work. When that person retired, things started going out of hand. Python ended up with complex eclasses that nobody wholly understood and that urgently needed updates. Many packages were getting outdated.

I believe that we did the best that we could at the time. We’ve started the new eclass set because it was much easier to draw a clear line and start over. We ended up removing many of the less important or more problematic packages, from almost the whole net-zope category (around 200 packages) to temporarily removing Django (only to readd it later on). Getting everything at least initially ported took a lot of time. Many of the ports turned out buggy, a large number of packages were outdated, unmaintained, missing test suites.

Today I can finally say that the Python team’s packages standing is reasonably good. However, in order for it to remain good we need to put a lot of effort every day. Packages need to be bumped. Bumps often add new dependencies. New Python versions require testing. Test regressions just keep popping up, and often when you’re working on something else. Stabilizations need to be regularly tracked, and they usually uncover even more obscure test problems. Right now it’s pretty normal for me to spend an hour of my time every day just to take care of the version bumps.

I have tried my best to avoid returning to a bus factor of one. I have tried to keep the eclasses simple (but I can’t call that a complete success), involve more people in the development, keep things well documented and packages clean. Yet I still end up doing a very large portion of work in the team. I know that there are other developers who can stand in for me but I’m not sure if they will be able to take up all the work, given all their other duties.

We really need young blood. We need people who would be able to dedicate a lot of time specifically to Python, and learn all the intricacies of Python packaging. While I’ve done my best to document everything I can think of in the Python Guide, it’s still very little. The Python ecosystem is a very diverse one, and surprisingly hard to maintain without burning out. Of course, there are many packages that are pleasure to maintain, and many upstreams that are pleasure to work with. Unfortunately, there are also many projects that make you really frustrated — with bad quality code, broken tests, lots of NIH dependencies and awful upstreams that simply hate packagers.

Over my 10 (and a half) years as a developer, I have done a lot of different things. However, if I were to point one area of Gentoo where I put most of my effort, that area would be Python. I am proud of all that we’ve been able to accomplish, and how great our team is. However, there are still many challenges ahead of us, as well as a lot of tedious work.

I’m a Gentoo developer for over 10 years already. I’ve been doing a lot of different things throughout that period. However, Python was pretty much always somewhere within my area of interest. I don’t really recall how it all started. Maybe it had something to do with Portage being written in Python. Maybe it was the natural next step after programming in Perl.

I feel like the upcoming switch to Python 3.9 is the last step in the prolonged effort of catching up with Python. Over the last years, we’ve been working real hard to move Python support forward, to bump neglected packages, to enable testing where tests are available, to test packages on new targets and unmask new targets as soon as possible. We have improved the processes a lot. Back when we were switching to Python 3.4, it took almost a year from the first false start attempt to the actual change. We started using Python 3.5 by default after upstream dropped bugfix support for it. In a month from now, we are going to start using Python 3.9 even before 3.10 final is released.

I think this is a great opportunity to look back and see what changed in the Gentoo Python ecosystem, in the last 10 years.

Python package ebuilds 10 years ago

Do you know how a Python package ebuild looked like 10 years ago? Let’s take gentoopm-0.1 as an example (reformatted to fit the narrow layout better):


# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.1.ebuild,v 1.1 2011/07/15 19:05:26 mgorny Exp $

EAPI=3

PYTHON_DEPEND='*:2.6'
SUPPORT_PYTHON_ABIS=1
RESTRICT_PYTHON_ABIS='2.4 2.5'
DISTUTILS_SRC_TEST=setup.py

inherit base distutils

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/gentoopm/"
SRC_URI="http://cloud.github.com/downloads/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.8.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="dev-python/epydoc"
PDEPEND="app-admin/eselect-package-manager"

src_prepare() {
  base_src_prepare
  distutils_src_prepare
}

src_compile() {
  distutils_src_compile

  if use doc; then
    "$(PYTHON -2)" setup.py doc || die
  fi
}

src_install() {
  distutils_src_install

  if use doc; then
    dohtml -r doc/* || die
  fi
}

This ebuild is actually using the newer API of python.eclass that is enabled via SUPPORT_PYTHON_ABIS. It provides support for installing for multiple implementations (like the modern python-r1 eclass). PYTHON_DEPEND is used to control the dependency string added to ebuild. The magical syntax here means that the ebuild supports both Python 2 and Python 3, from Python 2.6 upwards. RESTRICT_PYTHON_ABIS opts out support for Python versions prior to 2.6. Note the redundancy — PYTHON_DEPEND controls the dependency, specified as a range of Python 2 and/or Python 3 versions, RESTRICT_PYTHON_ABIS controls versions used at build time and needs to explicitly exclude all unsupported branches.

Back then, there were no PYTHON_TARGETS to control what was built. Instead, the eclass defaulted to using whatever was selected via eselect python, with the option to override it via setting USE_PYTHON in make.conf. Therefore, there were no cross-package USE dependencies and you had to run python-updater to verify whether all packages are built for the current interpreter, and rebuild these that were not.

Still, support for multiple ABIs, as the eclass called different branches/implementations of Python, was a major step forward. It was added around the time that the first releases of Python 3 were published, and our users have been relying on it to support a combination of Python 2 and Python 3 for years. Today, we’re primarily using it to aid developers in testing their packages and to provide a safer upgrade experience.

The python.eclass stalemate

Unfortunately, things at the time were not all that great. Late 2010 marks a conflict between the primary Python developer and the rest of the community, primarily due to the major breakage being caused by the changes in Python support. By mid-2011, it was pretty clear that there is no chance to resolve the conflict. The in-Gentoo version of python.eclass was failing to get EAPI 4 support for 6 months already, while an incompatible version continued being developed in the (old) Python overlay. As Dirkjan Ochtman related in his mail:

I guess by now pretty much everyone knows that the python eclass is rather complex, and that this poses some problems. This has also been an important cause for the disagreements between Arfrever and some of the other developers. Since it appears that Arfrever won’t be committing much code to gentoo-x86 in the near future, I’m trying to figure out where we should go with the python.eclass. […]

Dirkjan Ochtman, 2011-06-27, [gentoo-dev] The Python problem

Eventually, some of the changes from the Python overlay were backported to the eclass and EAPI 4 support was added. Nevertheless, at this point it was pretty clear that we need a new way forward. Unfortunately, the discussions were leading nowhere. With the primary eclass maintainer retired, nobody really comprehended most of the eclass, nor were able to afford the time to figure it out. At the same time, involved parties wanted to preserve backwards compatibility while moving forward.

The tie breaker: python-distutils-ng

Some of you might find it surprising that PYTHON_TARGETS are not really a python-r1 invention. Back in March 2012, when Python team was still unable to find a way forward with python.eclass, Krzysztof Pawlik (nelchael) has committed a new python-distutils-ng.eclass. It has never grown popular, and it has been replaced by the python-r1 suite before it ever started being a meaningful replacement for python.eclass. Still, it served an important impulse that made what came after possible.

Here’s a newer gentoopm ebuild using the new eclass (again reformatted):


# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-portage/gentoopm/gentoopm-0.2.5-r1.ebuild,v 1.1 2012/05/26 10:11:21 mgorny Exp $

EAPI=4
PYTHON_COMPAT='python2_6 python2_7 python3_1 python3_2'

inherit base python-distutils-ng

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/mgorny/gentoopm/"
SRC_URI="mirror://github/mgorny/${PN}/${P}.tar.bz2"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="~amd64 ~mips ~x86 ~x86-fbsd"
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/portage-2.1.10.3
    sys-apps/pkgcore
    >=sys-apps/paludis-0.64.2[python-bindings]
  )"
DEPEND="doc? ( dev-python/epydoc )"
PDEPEND="app-admin/eselect-package-manager"

python_prepare_all() {
  base_src_prepare
}

src_compile() {
  python-distutils-ng_src_compile
  if use doc; then
    "${PYTHON}" setup.py doc || die
  fi
}

python_install_all() {
  if use doc; then
    dohtml -r doc/*
  fi
}

Just looking at the code, you may see that python-r1 has inherited a lot after this eclass. python-distutils-ng in turn followed some of the good practices introduced before in the ruby-ng eclass. It introduced PYTHON_TARGETS to provide explicit visible control over implementations used for the build — though notably it did not include a way for packages to depend on matching flags (i.e. the equivalent of ${PYTHON_USEDEP}). It also used the sub-phase approach that makes distutils-r1 and ruby-ng eclasses much more convenient than the traditional python.eclass approach that roughly resembled using python_foreach_impl all the time.

What’s really important is that python-distutils-ng carved a way forward. It’s been a great inspiration and a proof of concept. It has shown that we do not have to preserve compatibility with python.eclass forever, or have to learn its inner workings before starting to solve problems. I can’t say how Python would look today if it did not happen but I can say with certainly that python-r1 would not happen so soon if it weren’t for it.

python-r1

In October 2012, the first version of python-r1 was committed. It combined some of the very good ideas of python-distutils-ng with some of my own.  The goal was not to provide an immediate replacement for python.eclass. Instead, the plan was to start simple and add new features as they turned out to be necessary. Not everything went perfectly but I dare say that the design has stood the test of time. While I feel like the eclasses ended up being more complex than I wished they would be, they still work fine with no replacement in sight and they serve as inspiration to other eclasses.

For completeness, here’s a 2017 gentoopm live ebuild that uses a pretty complete distutils-r1 feature set:


# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6
PYTHON_COMPAT=( python{2_7,3_4,3_5,3_6} pypy )

EGIT_REPO_URI="https://github.com/mgorny/gentoopm.git"
inherit distutils-r1 git-r3

DESCRIPTION="A common interface to Gentoo package managers"
HOMEPAGE="https://github.com/mgorny/gentoopm/"
SRC_URI=""

LICENSE="BSD-2"
SLOT="0"
KEYWORDS=""
IUSE="doc"

RDEPEND="
  || (
    >=sys-apps/pkgcore-0.9.4[${PYTHON_USEDEP}]
    >=sys-apps/portage-2.1.10.3[${PYTHON_USEDEP}]
    >=sys-apps/paludis-3.0.0_pre20170219[python,${PYTHON_USEDEP}]
  )"
DEPEND="
  doc? (
    dev-python/epydoc[$(python_gen_usedep python2_7)]
  )"
PDEPEND="app-eselect/eselect-package-manager"

REQUIRED_USE="
  doc? ( $(python_gen_useflags python2_7) )"

src_configure() {
  use doc && DISTUTILS_ALL_SUBPHASE_IMPLS=( python2.7 )
  distutils-r1_src_configure
}

python_compile_all() {
  use doc && esetup.py doc
}

python_test() {
  esetup.py test
}

python_install_all() {
  use doc && local HTML_DOCS=( doc/. )
  distutils-r1_python_install_all
}

The new eclasses have employed the same PYTHON_TARGETS flags for the general implementation choice but also added PYTHON_SINGLE_TARGET to make choosing the implementation more convenient (and predictable at the same time) when the package did not permit choosing more than one. The distutils-r1 eclass reused the great idea of sub-phases to make partial alterations to the phases easier.

The unique ideas included:

  • the split into more eclasses by functionality (vs switching the mode via variables as done in python.eclass)
  • exposing dependencies and REQUIRED_USE constraints via variables to be used in ebuild instead of writing elaborate mechanisms for adding dependencies
  • avoiding command substitution in global scope to keep metadata regeneration fast

The eclasses evolved a lot over the years. Some of the original ideas turned out pretty bad, e.g. trying to run sub-phases in parallel (which broke a lot of stuff for minor performance gain) or the horribly complex original interaction between PYTHON_TARGETS and PYTHON_SINGLE_TARGET. Ebuilds often missed dependencies and REQUIRED_USE constraints, until we finally made the pkgcheck-based CI report that.

The migration to new eclasses took many years. Initially, python-r1 ebuilds were even depending on python.eclass ebuilds. The old eclass was not removed until March 2017, i.e. 4.5 years after introducing the new eclasses.

Testing on new Python targets

We went for the opt-in approach with PYTHON_COMPAT. This means that for every new Python target added, we start with no packages supporting it and have to iterate over all of the packages adding the support. It’s a lot of work and it has repeatedly caused users pain due to packages not being ported in time for the big switch to the next version. Some people have complained about that and suggested that we should go for opt-out instead. However, if you think about it, opt-in is the only way to go.

The big deal is that for any particular package to support a new implementation, all of its Python dependencies need to support it as well. With the opt-in approach, it means that we’re doing the testing dependency-first, and reaching the bigger packages only when we confirm that there’s at least a single version of every dependency that works for it. If we do things right, users don’t even see any regressions.

If we went for the opt-out approach, all packages would suddenly claim to support the new version. Now, this wouldn’t be that bad if we were actually able to do a big CI run for all packages — but we can’t since a lot of them do not have proper tests, and Python version incompatibility often can’t be detected statically. In the end, we would be relying on someone (possibly an user) reporting that something is broken. Then we’d have to investigate where in the dependency chain the culprit is, and either restrict the new target (and then restrict it in all its reverse dependencies) or immediately fix it.

So in the end, opt-out would be worse for both users and developers. Users would hit package issues first hand, and developers would have to spend significant time on the back-and-forth effort of removing support for new targets, and then adding it again. If we are to add new targets early (which is a worthwhile goal), we have to expect incompatible packages. My experience so far shows that Gentoo developers sometimes end up being the first people to submit patches fixing the incompatibility. This can be a real problem given that many Python packages have slow release cycles, and are blocking their reverse dependencies, and these in turn block their reverse dependencies and so on.

Now, things did not always go smoothly. I have prepared a Python release and Gentoo packaging timeline that puts our packaging work into perspective. As you can see, we were always quite fast in packaging new Python interpreters but it took a significant time to actually switch the default targets to them — in fact, we often switched just before or even after upstream stopped providing bug fixes to the version in question.

Our approach has changed over the years. Early on, we generally kept both the interpreter and the target in ~arch, and stabilized them in order to switch targets. The prolonged stable-mask of the new target has resulted in inconsistent presence of support for the new target in stable, and this in turn involved a lot of last-minute stabilization work. Even then, we ended up switching targets before stable was really ready for that. This was particularly bad for Python 3.4 — the period seen as ‘stable’ on the timeline is actually a period following the first unsuccessful switch of the default. It took us over half a year to try again.

Then (around Python 3.6, if I’m not mistaken) we switched to a different approach. Instead of delaying till the expected switch, we’ve tried to unmask the target on stable systems as soon as possible. This way, we started enforcing dependency graph consistency earlier and were able to avoid big last minute stabilizations needed to unmask the target.

Eventually, thanks to pkgcheck’s superior StableRequestCheck, I’ve started proactively stabilizing new versions of Python packages. This was probably the most important improvement of all. The stabilization effort was streamlined, new versions of packages gained stable keywords sooner and along them did the support for new Python implementations.

The effort in package testing and stabilizations have finally made it possible to catch up with upstream. We have basically gone through a target switching sprint, moving through 3.7 and 3.8 in half a year each. The upcoming switch to Python 3.9 concludes this effort. For the first time in years, our default will be supported upstream for over 6 months.

That said, it is worth noting that things are not becoming easier for us. Over time, Python packages keep getting new dependencies, and this means that every new Python version will involve more and more porting work. Unfortunately, some high profile packages such as setuptools and pytest keep creating bigger and bigger dependency loops. At this point, it is no longer reasonable to attempt to port all the cyclic dependencies simultaneously. Instead, I tend to temporarily disable tests for the initial ports to reduce the number of dependencies. I’ve included a list of suggested initial steps in the Python Guide to ease future porting efforts.

Packaging the Python interpreter

CPython is not the worst thing to package but it’s not the easiest one either. Admittedly, the ebuilds were pretty good back when I joined. However, we’ve already carried a pretty large and mostly undocumented set of patches, and most of these patches we carry up to this day, with no chance of upstreaming them.

Some of the involved patches are build fixes and hacks that are either specific to Gentoo, or bring the flexibility Gentoo cares about (such as making some of the USE flags possible). There are also some fixes for old bugs that upstream has not shown any interest in fixing.

Another part of release management is resolving security bugs. Until recently, we did not track vulnerabilities in CPython very well. Thanks to the work of new Security team members, we have started being informed of vulnerabilities earlier. At the same time, we realized that CPython’s treatment of vulnerabilities is a bit suboptimal.

Admittedly, when very bad things happen upstream releases fixes quickly. However, non-critical vulnerability fixes are released as part of the normal release cycle, approximately every two months. For some time already, every new release contained some security fixes and had to be stabilized quickly. At the same time it contained many other changes with their own breakage potential. Old Python branches were even worse — absurdly, even though these versions received security fixes only, the releases were even rarer.

In the end, I’ve decided that it makes more sense to backport security fixes to our patchset as soon as I become aware of them, and stabilize the patch bumps instead. I do this even if upstream makes a new release at the same time, since patch bumps are safer stable targets. Even then, some of the security fixes actually require changing the behavior. To name a few recent changes:

  • urllib.parse.parse_qsl() historically used to split the URL query string on either & or ;. This somewhat surprising behavior was changed to split only on &, with a parameter to change the separator (but no option to restore the old behavior).
  • urllib.parse.urlparse() historically preserved newlines, CRs and tabs. In the latest Python versions this behavior was changed to follow a newer recommendation of stripping these characters. As a side effect, some URL validators (e.g. in Django) suddenly stopped rejecting URLs with newlines.
  • The ipaddress module recently stopped allowing leading zeros in IPv4 addresses. These were accepted before but some of the external libraries were incidentally interpreting them as octal numbers.

Some of these make you scratch your head.

Unsurprisingly, this is also a lot of work. At this very moment, we are maintaining six different slots of CPython (2.7, 3.6, 3.7, 3.8, 3.9, 3.10). For every security backport set, I start by identifying the security-related commits on the newest release branch. This is easy if they’re accompanied by a news entry in Security category — unfortunately, some vulnerability fixes were treated as regular bug fixes in the past. Once I have a list of commits, I cherry-pick them to our patched branch and make a patchset out of that. This is the easy part.

Now I have to iterate over all the older versions. For maintained branches, the first step is to identify if upstream has already backported their fixes. if they did, I cherry-pick the backport commits from the matching branch. If they did not, I need to verify if the vulnerability applies to this old version, and cherry-pick from a newer version. Sometimes it requires small changes.

The hardest part is Python 2.7 — it is not supported upstream but still used as a build-time dependency by a few projects. The standard library structure in Python 2 differs a lot from the one in Python 3 but many of Python 3 vulnerabilities stem from code dating back to Python 2. In the end, I have to find the relevant piece of original code in Python 2, see if it is vulnerable and usually rewrite the patch for the old code.

I wish this were the end. However, in Gentoo we are also packaging PyPy, the alternative Python implementation. PyPy follows its own release cycle. PyPy2 is based on now-unmaintained Python 2, so most of the vulnerability fixes from our Python 2.7 patchset need to be ported to it. PyPy3 tries to follow Python 3’s standard library but is not very fast at it. In the end, I end up cherry-picking the changes from CPython to PyPy for our patchsets, and at the same time sending them upstream’s way so that they can fix the vulnerabilities without having to sync with CPython immediately.

Historically, we have also supported another alternative implementation, Jython. However, for a very long time upstream’s been stuck on Python 2.5 and we’ve eventually dropped support for Jython as it became unmaintainable. Upstream has eventually caught up with Python 2.7 but you can imagine how useful that is today.

There are other interesting projects out there but I don’t think we have the manpower (or even a very good reason) to work on them.

Now, a fun fact: Victor Stinner indicates in his mail Need help to fix known Python security vulnerabilities that the security fixes CPython receives are only a drop in the ocean. Apparently, at the time of writing there were 78 open problems of various severity.

A bus and a phoenix

Any project where a single developer does a sufficiently large portion of work has a tendency towards a bus factor of one. The Python project was no different. Ten years ago we were relying on a single person doing most of the work. When that person retired, things started going out of hand. Python ended up with complex eclasses that nobody wholly understood and that urgently needed updates. Many packages were getting outdated.

I believe that we did the best that we could at the time. We’ve started the new eclass set because it was much easier to draw a clear line and start over. We ended up removing many of the less important or more problematic packages, from almost the whole net-zope category (around 200 packages) to temporarily removing Django (only to readd it later on). Getting everything at least initially ported took a lot of time. Many of the ports turned out buggy, a large number of packages were outdated, unmaintained, missing test suites.

Today I can finally say that the Python team’s packages standing is reasonably good. However, in order for it to remain good we need to put a lot of effort every day. Packages need to be bumped. Bumps often add new dependencies. New Python versions require testing. Test regressions just keep popping up, and often when you’re working on something else. Stabilizations need to be regularly tracked, and they usually uncover even more obscure test problems. Right now it’s pretty normal for me to spend an hour of my time every day just to take care of the version bumps.

I have tried my best to avoid returning to a bus factor of one. I have tried to keep the eclasses simple (but I can’t call that a complete success), involve more people in the development, keep things well documented and packages clean. Yet I still end up doing a very large portion of work in the team. I know that there are other developers who can stand in for me but I’m not sure if they will be able to take up all the work, given all their other duties.

We really need young blood. We need people who would be able to dedicate a lot of time specifically to Python, and learn all the intricacies of Python packaging. While I’ve done my best to document everything I can think of in the Python Guide, it’s still very little. The Python ecosystem is a very diverse one, and surprisingly hard to maintain without burning out. Of course, there are many packages that are pleasure to maintain, and many upstreams that are pleasure to work with. Unfortunately, there are also many projects that make you really frustrated — with bad quality code, broken tests, lots of NIH dependencies and awful upstreams that simply hate packagers.

Over my 10 (and a half) years as a developer, I have done a lot of different things. However, if I were to point one area of Gentoo where I put most of my effort, that area would be Python. I am proud of all that we’ve been able to accomplish, and how great our team is. However, there are still many challenges ahead of us, as well as a lot of tedious work.

March 26 2021

LetsEncrypt SSL certificates for vhosts within the mail stack – via Postfix (SMTP) and Dovecot (IMAP)

Nathan Zachary (nathanzachary) March 26, 2021, 4:36

For a long time, I didn’t care about using self-signed SSL certificates for the mail stack because 1) they still secured the connection to the server, and 2) those certificates weren’t seen or utilised by anyone other than me. However, for my datacentre infrastructure which houses clients’ websites and email, using self-signed or generic certificates (even for the mail stack) wasn’t a very good solution as mail clients (e.g Thunderbird or Outlook) notify users of the problematic certs. Clients with dedicated mail servers could use valid certificates (freely from LetsEncrypt) without problem, but those on shared infrastructure posed a different issue—how can mail for different domains all sharing the same IPv4 address use individualised certificates for their SMTP and IMAP connections? This article will explain the method that I used to assign domain-specific certificates for the full mail stack using LetsEncrypt’s certbot for the certs themselves, the Postfix MTA (mail transfer agent [for SMTP]), and the Dovecot IMAP server. This article is tailored to Gentoo Linux, but should be easily applied to nearly any distribution. Two general caveats are:

  • The locations of files may be different for your distribution. Consult your distribution’s documentation for the appropriate file locations
  • I use OpenRC as my init system in Gentoo. If your distribution uses systemd, you will need to replace any reference to code snippets containing /etc/init.d/$SERVICE $ACTION with systemctl $ACTION $SERVICE
    • As an example, I mention restarting Dovecot with /etc/init.d/dovecot restart
    • A systemd user would instead issue systemctl restart dovecot

I will provide a significant amount of ancillary information pertaining to each step of this process. If you are comfortable with the mail stack, some of it may be rudimentary, so feel free to skip ahead. Conversely, if any of the concepts are new or foreign to you, please reference Gentoo’s documentation for setting up a Complete Virtual Mail Server as a prerequisite. For the remainder of the article, I am going to use two hypothetical domains of domain1.com and domain2.com. Any time that they are referenced, you will want to replace them with your actual domains.

BIND (DNS) configurations

In order to keep the web stack and mail stack separate in terms of DNS, I like to have mail.domain1.com & mail.domain2.com subdomains for the MX records, but just have them point to the same A record used for the website. Some may consider this to be unnecessary, but I have found the separation helpful for troubleshooting if any problems should arise. Here are the relevant portions of the zone files for each domain:

# grep -e 'A\|MX ' /etc/bind/pri/domain1.com.external.zone 
domain1.com.		300	IN	A	$IP_ADDRESS
mail.domain1.com.	300	IN	A	$IP_ADDRESS

# grep -e 'A\|MX ' /etc/bind/pri/domain2.com.external.zone 
domain2.com.		300	IN	A	$IP_ADDRESS
mail.domain2.com.	300	IN	A	$IP_ADDRESS

In the above snippets, $IP_ADDRESS should be the actual IPv4 address of the webserver. It should be noted that, in this setup, the web stack and the mail stack reside on the same physical host, so the IP is the same for both stacks.

Apache (webserver) configurations

As mentioned above, I keep the web stack and mail stack separate in terms of DNS. For the LetsEncrypt certificates (covered in the next section), though, I use the same certificate for both stacks. I do so by generating the cert for both the main domain and the ‘mail’ subdomain. In order for this to work, I make the ‘mail’ subdomain a ServerAlias in the Apache vhost configurations:

# grep -e 'ServerName \|ServerAlias ' www.domain1.com.conf 
	ServerName domain1.com
	ServerAlias www.domain1.com mail.domain1.com

# grep -e 'ServerName \|ServerAlias ' www.domain2.com.conf 
	ServerName domain2.com
	ServerAlias www.domain2.com mail.domain2.com

This allows the verification of the ‘mail’ subdomain to be done via the main URL instead of requiring a separate public-facing site directory for it.

LetsEncrypt SSL certificates

LetsEncrypt is a non-profit certificate authority (CA) that provides X.509 (TLS) certificates free-of-charge. The issued certificates are only valid for 90 days, which encourages automated processes to handle renewals. The recommended method is to use the certbot tool for renewals, and there are many plugins available that provide integration with various webservers. Though I run a combination of Apache and NGINX, I prefer to not have certbot directly interact with them. Rather, I choose to rely on certbot solely for the certificate generation & renewal, and to handle the installation thereof via other means. For this tutorial, I will use the ‘certonly’ option with the webroot plugin:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/$DOMAIN/$HOST/htdocs/ --domains $DOMAIN,$DOMAIN,$DOMAIN

In the code snippet above, you will replace $DOMAIN with the actual domain, and $HOST with the subdomain. So, for our two hypothetical domains, the commands translate as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,mail.domain1.com

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain2.com/www/htdocs/ --domains domain2.com,www.domain2.com,mail.domain2.com

The webroot plugin will create a temporary file under ${webroot-path}/.well-known/acme-challenge/ and then check that file via HTTP in order to validate the server. Make sure that the directory is publicly accessible or else the validation will fail. Once certbot validates the listed domains—in this setup, the ‘www’ and ‘mail’ subdomains are just aliases to the primary domain (see the BIND and Apache configurations sections above)—it will generate the SSL certificates and place them under /etc/letsencrypt/live/domain1.com/ and /etc/letsencrypt/live/domain2.com/, respectively. There will be two files for each certificate:

  • fullchain.pem –> the public certificate
  • privkey.pem –> the private key for the certificate

Though these certificates are often used in conjunction with the web stack, we are going to use them for securing the mail stack as well.

Dovecot (IMAP) configurations

Now that we have the certificates for each domain, we’ll start by securing the IMAP server (i.e. Dovecot) so that the users’ Mail User Agent (MUA, or more colloquially, “email client” [like Thunderbird or Outlook]) will no longer require a security exception due to a domain mismatch. Adding the domain-specific SSL certificate to Dovecot is a straightforward process that only requires two directives per domain. For domain1.com and domain2.com, add the following lines to /etc/dovecot/conf.d/10-ssl.conf:

local_name mail.domain1.com {
  ssl_cert = </etc/letsencrypt/live/domain1.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain1.com/privkey.pem
}

local_name mail.domain2.com {
  ssl_cert = </etc/letsencrypt/live/domain2.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain2.com/privkey.pem
}

Those code blocks can be copied and pasted for any additional virtual hosts or domains that are needed. As with any configuration change, make sure to restart the application in order to make the changes active:

/etc/init.d/dovecot restart
Postfix (SMTP) configurations

Configuring Dovecot to use the SSL certificate for securing the IMAP connection from the user’s email client to the server is only one part of the process—namely the connection when the user is retrieving mails from the server. This next part will use the same certificate to secure the SMTP connection (via the Postfix SMTP server) for sending mails.

The first step is to create a file that will be used for mapping each certificate to its respective domain. Postfix can handle this correlation via Server Name Indication (SNI), which is an extension of the TLS protocol that indicates the hostname of the server at the beginning of the handshake process. Though there is no naming requirement for this map file, I chose to create it as /etc/postfix/vmail_ssl. The format of the file is:

$DOMAIN   $PRIVATE_KEY   $PUBLIC_KEY_CHAIN

So for our example of domain1.com and domain2.com, the file would consist of the following entries:

mail.domain1.com /etc/letsencrypt/live/domain1.com/privkey.pem /etc/letsencrypt/live/domain1.com/fullchain.pem

mail.domain2.com /etc/letsencrypt/live/domain2.com/privkey.pem /etc/letsencrypt/live/domain2.com/fullchain.pem

Though this file is in plaintext, Postfix doesn’t understand the mapping in this format. Instead, a base64-encoded Berkeley DB file needs to be used. Thankfully, Postfix makes creating such a file very easy via the postmap utility. Once you have created and populated your /etc/postfix/vmail_ssl file with the entries for each domain, issue the following command:

# postmap -F hash:/etc/postfix/vmail_ssl

which will create the Berkeley DB file (named vmail_ssl.db) in the same directory:

# find /etc/postfix/ -type f -iname '*vmail_ssl*'
/etc/postfix/vmail_ssl.db
/etc/postfix/vmail_ssl

# file /etc/postfix/vmail_ssl
/etc/postfix/vmail_ssl: ASCII text

# file /etc/postfix/vmail_ssl.db 
/etc/postfix/vmail_ssl.db: Berkeley DB (Hash, version 10, native byte-order)

Now that we have created the mapping table, we have to configure Postfix to use it for SMTP connections. It’s acceptable to have both the SNI-mapped certificates AND a generic SSL certificate as the default (for when a domain isn’t listed in the mapping table). Postfix can have both directives specified simultaneously. To do so, the following directives need to be added to /etc/postfix/main.cf (the comments explain both sets of directives):

## Default SSL cert for SMTP if SNI is not enabled
smtpd_tls_cert_file = /etc/ssl/mail/server.pem
smtpd_tls_key_file = /etc/ssl/mail/server.key

## Mappings for SMTP SSL certs when SNI is enabled
tls_server_sni_maps = hash:/etc/postfix/vmail_ssl

After making those modifications to Postfix’s main configuration file, it is required to restart it:

/etc/init.d/postfix restart

That’s it! Now the full mail stack is secured using domain-specific SSL certificates for both IMAP and SMTP connections. The remaining sections below will explain some maintenance-related procedures such as handling the LetsEncrypt certificate renewals & updating the mappings in Postfix automatically, as well as verifying it’s all working as intended (and some troubleshooting tips in case it’s not). ♦

Automatic renewal (cron) configurations

As mentioned in the LetsEncrypt section above, the certificates that they issue are only valid for a period of 90 days. One of the reasons for the relatively short validity period is to encourage automation when it comes to renewing them. I choose to handle the renewals automatically via cron:

# tail -n 4 /var/spool/cron/crontabs/root 

## LetsEncrypt certificate renewals on first of each month
## See /etc/letsencrypt/renewal-hooks/post/ for Postfix & Apache hooks 
0  2  1  *  *   /usr/bin/certbot renew --quiet

This cron entry instructs LetsEncrypt’s certbot to check the validity of ALL certificates at 02:00 (server time) on the first of every month (if that format is unfamiliar to you, see Wikipedia’s article on cron). The renew subcommand will automatically generate a new certificate for any found to expire within the next 30 days, and the quiet option will silence any output except for errors, which is appropriate for use with a cron job.

That’s the procedure for renewing the certificate automatically, but what about then automatically updating the appropriate stack configurations—in particular, Postfix’s vmail_ssl mappings table (and Apache, but that’s outside the scope of this tutorial)? If the certificate is renewed, but it is not updated in Postfix’s hash table, there will be a mismatch error. As mentioned in the comment on the cron entry, I chose to handle those configuration updates automatically via certbot’s ‘renewal hooks’, which can be found under /etc/letsencrypt/renewal-hooks/. In this case, the configuration updates need to happen after certificate renewal, so they are put under the post/ subdirectory.

I have two scripts that run after a certificate renewal, but only the 01_postfix_smtp_ssl.sh one is applicable for the mail stack:

# ls /etc/letsencrypt/renewal-hooks/post/
01_postfix_smtp_ssl.sh  02_apache.sh

# cat /etc/letsencrypt/renewal-hooks/post/01_postfix_smtp_ssl.sh 
#!/bin/bash
/usr/sbin/postmap -F hash:/etc/postfix/vmail_ssl
/etc/init.d/postfix restart
exit 0

The simple script issues the same postmap command from the ‘Postfix (SMTP) configurations‘ section above, and then restarts Postfix. If everything goes smoothly, it will exit cleanly (‘exit 0’). The script ensures that the new certificate is immediately applied to the Postfix configuration so that there aren’t validation errors after the automated renewal process.

Verification of the certificates

If everything went according to plan, valid SSL certificates should be in place for both mail.domain1.com and mail.domain2.com. Like any good engineer, though, we don’t want to just assume that it’s working as intended. So… we should test it! You could just open an email client of your choice and view the certificates for IMAP and SMTP connections. Personally, though, I prefer using terminal-based utilities as I find them to be more efficient. In this case, we can use the openssl command for connecting to each domain as a test, and the basic syntax is:

For SMTP:

openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp

For IMAP:

openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com

These commands will output a lot of information including the full public certificate, the issuing authority (LetsEncrypt), handshake details, SSL session details, and so on. If you’re interested in all of those details, feel free to issue the commands as they are above (obviously swapping out the actual domains and the ports that you use for SMTP and IMAP). If, however, you simply want to confirm that the certificates are valid, you can pipe the commands to grep in order to limit the output:

For SMTP:

$ openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
250 CHUNKING
subject=CN = domain1.com
Verify return code: 0 (ok)

For IMAP:

$ openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
subject=CN = domain1.com
Verify return code: 0 (ok)

If you see output similar to what’s above, then everything is working as it should be. In particular, you want to make sure that references to the ‘CN’ match the domain, and that you see a ‘Verify return code:’ of 0 (ok) Pat yourself on the back and grab your beverage of choice to celebrate a job well done. ♦

Additional information

If you have already been using a domain for a website or other service, chances are that you have already generated a LetsEncrypt SSL certificate for it. Thankfully LetsEncrypt makes it easy to append a new subdomain to an existing certificate instead of having to generate a completely separate one for the ‘mail’ subdomain used in this guide (e.g. mail.domain1.com).

The first step is to find the certificate that you want to modify (in this case, domain1.com) and see which subdomains are covered under it. This can be accomplished using the certbot certificates command. The output will look something like this:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com
   Expiry Date: 2021-05-02 06:03:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

The important part is the list of subdomains on the Domains: line because you need to reference ALL of them when using the --expand flag that follows. Using the output from above, the command would be constructed as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,staging.domain1.com,mail.domain1.com --expand

If certbot indicates that the new certificate has been generated without any errors, you can check it again using the certbot certificates command from above and validate that now the ‘mail’ subdomain is listed as well:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com mail.domain1.com
   Expiry Date: 2021-05-14 06:05:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem
Troubleshooting

I can’t anticipate the full gamut of problems that could potentially arise when going through this guide, but I will try to cover some common pitfalls here. If you run into a problem, feel free to comment and I will try to help you through it.

>>> Postfix error about the table hash:

If Postfix won’t start after the modifications from the sections above, and you see a line like this in the mail logs:

[postfix/smtpd] warning: table hash:/etc/postfix/vmail_ssl.db: key mail.domain1.com: malformed BASE64 value: /etc/letsencrypt/live/domain1

then the problem stems from running postmap without the -F flag. Try it again with that flag: postmap -F hash:/etc/postfix/vmail_ssl which should create a syntactically correct hash table, allowing Postfix to properly start up.

For a long time, I didn’t care about using self-signed SSL certificates for the mail stack because 1) they still secured the connection to the server, and 2) those certificates weren’t seen or utilised by anyone other than me. However, for my datacentre infrastructure which houses clients’ websites and email, using self-signed or generic certificates (even for the mail stack) wasn’t a very good solution as mail clients (e.g Thunderbird or Outlook) notify users of the problematic certs. Clients with dedicated mail servers could use valid certificates (freely from LetsEncrypt) without problem, but those on shared infrastructure posed a different issue—how can mail for different domains all sharing the same IPv4 address use individualised certificates for their SMTP and IMAP connections? This article will explain the method that I used to assign domain-specific certificates for the full mail stack using LetsEncrypt’s certbot for the certs themselves, the Postfix MTA (mail transfer agent [for SMTP]), and the Dovecot IMAP server. This article is tailored to Gentoo Linux, but should be easily applied to nearly any distribution. Two general caveats are:

  • The locations of files may be different for your distribution. Consult your distribution’s documentation for the appropriate file locations
  • I use OpenRC as my init system in Gentoo. If your distribution uses systemd, you will need to replace any reference to code snippets containing /etc/init.d/$SERVICE $ACTION with systemctl $ACTION $SERVICE
    • As an example, I mention restarting Dovecot with /etc/init.d/dovecot restart
    • A systemd user would instead issue systemctl restart dovecot

I will provide a significant amount of ancillary information pertaining to each step of this process. If you are comfortable with the mail stack, some of it may be rudimentary, so feel free to skip ahead. Conversely, if any of the concepts are new or foreign to you, please reference Gentoo’s documentation for setting up a Complete Virtual Mail Server as a prerequisite. For the remainder of the article, I am going to use two hypothetical domains of domain1.com and domain2.com. Any time that they are referenced, you will want to replace them with your actual domains.

BIND (DNS) configurations

In order to keep the web stack and mail stack separate in terms of DNS, I like to have mail.domain1.com & mail.domain2.com subdomains for the MX records, but just have them point to the same A record used for the website. Some may consider this to be unnecessary, but I have found the separation helpful for troubleshooting if any problems should arise. Here are the relevant portions of the zone files for each domain:

# grep -e 'A\|MX ' /etc/bind/pri/domain1.com.external.zone 
domain1.com.		300	IN	A	$IP_ADDRESS
mail.domain1.com.	300	IN	A	$IP_ADDRESS

# grep -e 'A\|MX ' /etc/bind/pri/domain2.com.external.zone 
domain2.com.		300	IN	A	$IP_ADDRESS
mail.domain2.com.	300	IN	A	$IP_ADDRESS

In the above snippets, $IP_ADDRESS should be the actual IPv4 address of the webserver. It should be noted that, in this setup, the web stack and the mail stack reside on the same physical host, so the IP is the same for both stacks.

Apache (webserver) configurations

As mentioned above, I keep the web stack and mail stack separate in terms of DNS. For the LetsEncrypt certificates (covered in the next section), though, I use the same certificate for both stacks. I do so by generating the cert for both the main domain and the ‘mail’ subdomain. In order for this to work, I make the ‘mail’ subdomain a ServerAlias in the Apache vhost configurations:

# grep -e 'ServerName \|ServerAlias ' www.domain1.com.conf 
	ServerName domain1.com
	ServerAlias www.domain1.com mail.domain1.com

# grep -e 'ServerName \|ServerAlias ' www.domain2.com.conf 
	ServerName domain2.com
	ServerAlias www.domain2.com mail.domain2.com

This allows the verification of the ‘mail’ subdomain to be done via the main URL instead of requiring a separate public-facing site directory for it.

LetsEncrypt SSL certificates

LetsEncrypt is a non-profit certificate authority (CA) that provides X.509 (TLS) certificates free-of-charge. The issued certificates are only valid for 90 days, which encourages automated processes to handle renewals. The recommended method is to use the certbot tool for renewals, and there are many plugins available that provide integration with various webservers. Though I run a combination of Apache and NGINX, I prefer to not have certbot directly interact with them. Rather, I choose to rely on certbot solely for the certificate generation & renewal, and to handle the installation thereof via other means. For this tutorial, I will use the ‘certonly’ option with the webroot plugin:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/$DOMAIN/$HOST/htdocs/ --domains $DOMAIN,$DOMAIN,$DOMAIN

In the code snippet above, you will replace $DOMAIN with the actual domain, and $HOST with the subdomain. So, for our two hypothetical domains, the commands translate as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,mail.domain1.com

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain2.com/www/htdocs/ --domains domain2.com,www.domain2.com,mail.domain2.com

The webroot plugin will create a temporary file under ${webroot-path}/.well-known/acme-challenge/ and then check that file via HTTP in order to validate the server. Make sure that the directory is publicly accessible or else the validation will fail. Once certbot validates the listed domains—in this setup, the ‘www’ and ‘mail’ subdomains are just aliases to the primary domain (see the BIND and Apache configurations sections above)—it will generate the SSL certificates and place them under /etc/letsencrypt/live/domain1.com/ and /etc/letsencrypt/live/domain2.com/, respectively. There will be two files for each certificate:

  • fullchain.pem –> the public certificate
  • privkey.pem –> the private key for the certificate

Though these certificates are often used in conjunction with the web stack, we are going to use them for securing the mail stack as well.

Dovecot (IMAP) configurations

Now that we have the certificates for each domain, we’ll start by securing the IMAP server (i.e. Dovecot) so that the users’ Mail User Agent (MUA, or more colloquially, “email client” [like Thunderbird or Outlook]) will no longer require a security exception due to a domain mismatch. Adding the domain-specific SSL certificate to Dovecot is a straightforward process that only requires two directives per domain. For domain1.com and domain2.com, add the following lines to /etc/dovecot/conf.d/10-ssl.conf:

local_name mail.domain1.com {
  ssl_cert = </etc/letsencrypt/live/domain1.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain1.com/privkey.pem
}

local_name mail.domain2.com {
  ssl_cert = </etc/letsencrypt/live/domain2.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain2.com/privkey.pem
}

Those code blocks can be copied and pasted for any additional virtual hosts or domains that are needed. As with any configuration change, make sure to restart the application in order to make the changes active:

/etc/init.d/dovecot restart

Postfix (SMTP) configurations

Configuring Dovecot to use the SSL certificate for securing the IMAP connection from the user’s email client to the server is only one part of the process—namely the connection when the user is retrieving mails from the server. This next part will use the same certificate to secure the SMTP connection (via the Postfix SMTP server) for sending mails.

The first step is to create a file that will be used for mapping each certificate to its respective domain. Postfix can handle this correlation via Server Name Indication (SNI), which is an extension of the TLS protocol that indicates the hostname of the server at the beginning of the handshake process. Though there is no naming requirement for this map file, I chose to create it as /etc/postfix/vmail_ssl. The format of the file is:

$DOMAIN   $PRIVATE_KEY   $PUBLIC_KEY_CHAIN

So for our example of domain1.com and domain2.com, the file would consist of the following entries:

mail.domain1.com /etc/letsencrypt/live/domain1.com/privkey.pem /etc/letsencrypt/live/domain1.com/fullchain.pem

mail.domain2.com /etc/letsencrypt/live/domain2.com/privkey.pem /etc/letsencrypt/live/domain2.com/fullchain.pem

Though this file is in plaintext, Postfix doesn’t understand the mapping in this format. Instead, a base64-encoded Berkeley DB file needs to be used. Thankfully, Postfix makes creating such a file very easy via the postmap utility. Once you have created and populated your /etc/postfix/vmail_ssl file with the entries for each domain, issue the following command:

# postmap -F hash:/etc/postfix/vmail_ssl

which will create the Berkeley DB file (named vmail_ssl.db) in the same directory:

# find /etc/postfix/ -type f -iname '*vmail_ssl*'
/etc/postfix/vmail_ssl.db
/etc/postfix/vmail_ssl

# file /etc/postfix/vmail_ssl
/etc/postfix/vmail_ssl: ASCII text

# file /etc/postfix/vmail_ssl.db 
/etc/postfix/vmail_ssl.db: Berkeley DB (Hash, version 10, native byte-order)

Now that we have created the mapping table, we have to configure Postfix to use it for SMTP connections. It’s acceptable to have both the SNI-mapped certificates AND a generic SSL certificate as the default (for when a domain isn’t listed in the mapping table). Postfix can have both directives specified simultaneously. To do so, the following directives need to be added to /etc/postfix/main.cf (the comments explain both sets of directives):

## Default SSL cert for SMTP if SNI is not enabled
smtpd_tls_cert_file = /etc/ssl/mail/server.pem
smtpd_tls_key_file = /etc/ssl/mail/server.key

## Mappings for SMTP SSL certs when SNI is enabled
tls_server_sni_maps = hash:/etc/postfix/vmail_ssl

After making those modifications to Postfix’s main configuration file, it is required to restart it:

/etc/init.d/postfix restart

That’s it! Now the full mail stack is secured using domain-specific SSL certificates for both IMAP and SMTP connections. The remaining sections below will explain some maintenance-related procedures such as handling the LetsEncrypt certificate renewals & updating the mappings in Postfix automatically, as well as verifying it’s all working as intended (and some troubleshooting tips in case it’s not). 🙂

Automatic renewal (cron) configurations

As mentioned in the LetsEncrypt section above, the certificates that they issue are only valid for a period of 90 days. One of the reasons for the relatively short validity period is to encourage automation when it comes to renewing them. I choose to handle the renewals automatically via cron:

# tail -n 4 /var/spool/cron/crontabs/root 

## LetsEncrypt certificate renewals on first of each month
## See /etc/letsencrypt/renewal-hooks/post/ for Postfix & Apache hooks 
0  2  1  *  *   /usr/bin/certbot renew --quiet

This cron entry instructs LetsEncrypt’s certbot to check the validity of ALL certificates at 02:00 (server time) on the first of every month (if that format is unfamiliar to you, see Wikipedia’s article on cron). The renew subcommand will automatically generate a new certificate for any found to expire within the next 30 days, and the quiet option will silence any output except for errors, which is appropriate for use with a cron job.

That’s the procedure for renewing the certificate automatically, but what about then automatically updating the appropriate stack configurations—in particular, Postfix’s vmail_ssl mappings table (and Apache, but that’s outside the scope of this tutorial)? If the certificate is renewed, but it is not updated in Postfix’s hash table, there will be a mismatch error. As mentioned in the comment on the cron entry, I chose to handle those configuration updates automatically via certbot’s ‘renewal hooks’, which can be found under /etc/letsencrypt/renewal-hooks/. In this case, the configuration updates need to happen after certificate renewal, so they are put under the post/ subdirectory.

I have two scripts that run after a certificate renewal, but only the 01_postfix_smtp_ssl.sh one is applicable for the mail stack:

# ls /etc/letsencrypt/renewal-hooks/post/
01_postfix_smtp_ssl.sh  02_apache.sh

# cat /etc/letsencrypt/renewal-hooks/post/01_postfix_smtp_ssl.sh 
#!/bin/bash
/usr/sbin/postmap -F hash:/etc/postfix/vmail_ssl
/etc/init.d/postfix restart
exit 0

The simple script issues the same postmap command from the ‘Postfix (SMTP) configurations‘ section above, and then restarts Postfix. If everything goes smoothly, it will exit cleanly (‘exit 0’). The script ensures that the new certificate is immediately applied to the Postfix configuration so that there aren’t validation errors after the automated renewal process.

Verification of the certificates

If everything went according to plan, valid SSL certificates should be in place for both mail.domain1.com and mail.domain2.com. Like any good engineer, though, we don’t want to just assume that it’s working as intended. So… we should test it! You could just open an email client of your choice and view the certificates for IMAP and SMTP connections. Personally, though, I prefer using terminal-based utilities as I find them to be more efficient. In this case, we can use the openssl command for connecting to each domain as a test, and the basic syntax is:

For SMTP:

openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp

For IMAP:

openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com

These commands will output a lot of information including the full public certificate, the issuing authority (LetsEncrypt), handshake details, SSL session details, and so on. If you’re interested in all of those details, feel free to issue the commands as they are above (obviously swapping out the actual domains and the ports that you use for SMTP and IMAP). If, however, you simply want to confirm that the certificates are valid, you can pipe the commands to grep in order to limit the output:

For SMTP:

$ openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
250 CHUNKING
subject=CN = domain1.com
Verify return code: 0 (ok)

For IMAP:

$ openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
subject=CN = domain1.com
Verify return code: 0 (ok)

If you see output similar to what’s above, then everything is working as it should be. In particular, you want to make sure that references to the ‘CN’ match the domain, and that you see a ‘Verify return code:’ of 0 (ok) Pat yourself on the back and grab your beverage of choice to celebrate a job well done. 🙂

Additional information

If you have already been using a domain for a website or other service, chances are that you have already generated a LetsEncrypt SSL certificate for it. Thankfully LetsEncrypt makes it easy to append a new subdomain to an existing certificate instead of having to generate a completely separate one for the ‘mail’ subdomain used in this guide (e.g. mail.domain1.com).

The first step is to find the certificate that you want to modify (in this case, domain1.com) and see which subdomains are covered under it. This can be accomplished using the certbot certificates command. The output will look something like this:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com
   Expiry Date: 2021-05-02 06:03:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

The important part is the list of subdomains on the Domains: line because you need to reference ALL of them when using the --expand flag that follows. Using the output from above, the command would be constructed as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,staging.domain1.com,mail.domain1.com --expand

If certbot indicates that the new certificate has been generated without any errors, you can check it again using the certbot certificates command from above and validate that now the ‘mail’ subdomain is listed as well:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com mail.domain1.com
   Expiry Date: 2021-05-14 06:05:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

Troubleshooting

I can’t anticipate the full gamut of problems that could potentially arise when going through this guide, but I will try to cover some common pitfalls here. If you run into a problem, feel free to comment and I will try to help you through it.

>>> Postfix error about the table hash:

If Postfix won’t start after the modifications from the sections above, and you see a line like this in the mail logs:

[postfix/smtpd] warning: table hash:/etc/postfix/vmail_ssl.db: key mail.domain1.com: malformed BASE64 value: /etc/letsencrypt/live/domain1

then the problem stems from running postmap without the -F flag. Try it again with that flag: postmap -F hash:/etc/postfix/vmail_ssl which should create a syntactically correct hash table, allowing Postfix to properly start up.

March 14 2021

Gentoo AMD64 Handbook "Preparing the disks" section reworked

Andreas K. Hüttel (dilfridge) March 14, 2021, 9:55

♦Since the text was becoming more and more outdated and also more and more convoluted, I have completely reworked the "Preparing the disks" section of the Gentoo AMD64 handbook. 

  • Since fdisk supports GUID partition tables (GPT) for a long time now, references to parted have been dropped.
  • The text restricts itself now to the combinations 1) UEFI boot and GPT and 2) BIOS / legacy boot and MBR. While mixing and matching is here certainly possible, we should treat it out of the scope of the manual.
  • Hopefully the terminology regarding the boot partition, UEFI system partition, and BIOS boot partition is more clear now (it was horribly mixed up before).

Please proofread and check for mistakes! I'll drop the "work in progress" label in a few days if nothing comes up.

Since the text was becoming more and more outdated and also more and more convoluted, I have completely reworked the "Preparing the disks" section of the Gentoo AMD64 handbook

  • Since fdisk supports GUID partition tables (GPT) for a long time now, references to parted have been dropped.
  • The text restricts itself now to the combinations 1) UEFI boot and GPT and 2) BIOS / legacy boot and MBR. While mixing and matching is here certainly possible, we should treat it out of the scope of the manual.
  • Hopefully the terminology regarding the boot partition, UEFI system partition, and BIOS boot partition is more clear now (it was horribly mixed up before).

Please proofread and check for mistakes! I'll drop the "work in progress" label in a few days if nothing comes up.

March 03 2021

Moving commits between independent git histories

Michał Górny (mgorny) March 03, 2021, 17:55

PyPy is an alternative Python implementation. While it does replace a large part of the interpreter, a large part of the standard library is shared with CPython. As a result, PyPy is frequently affected by the same vulnerabilities as CPython, and we have to backport security fixes to it.

Backporting security fixes inside CPython is relatively easy. All main Python branches are in a single repository, so it’s just a matter of cherry-picking the commits. Normally, you can easily move patches between two related git repositories using git-style patches but this isn’t going to work for two repositories with unrelated histories.

Does this mean manually patching PyPy and rewriting commit messages by hand? Luckily, there’s a relatively simple git am trick that can help you avoid that.

Roughly, the idea is to:

1. Create a git-format-patch of the change to backport.

2. Attempt to apply the change via git am — it will fail and your repository will be left in middle of an am session.

3. Apply the change via patch.

4. git add the changes.

5. Finally, call git am --continue to finish, wrapping your changes in the original commit metadata.

For example, let’s try backporting CVE-2021-23336 (parameter cloaking) fix:

First, grab the relevant patch from the CPython repository:

$ git format-patch -1 d0d4d30882fe3ab9b1badbecf5d15d94326fd13e
0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch

Then, inside the local clone of a random PyPy git mirror:

$ git am -3 ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
error: sha1 information is lacking or useless (Doc/library/cgi.rst).
error: could not build fake ancestor
Patch failed at 0001 bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Now enter the directory that stdlib resides in, and apply the patch manually, skipping any missing files:

$ patch -p2 < ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
can't find file to patch at input line 39
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|From d0d4d30882fe3ab9b1badbecf5d15d94326fd13e Mon Sep 17 00:00:00 2001
|From: Senthil Kumaran 
|Date: Mon, 15 Feb 2021 10:34:14 -0800
|Subject: [PATCH] [3.7] bpo-42967: only use '&' as a query string separator
| (GH-24297)  (GH-24531)
|MIME-Version: 1.0
|Content-Type: text/plain; charset=UTF-8
|Content-Transfer-Encoding: 8bit
|
|bpo-42967: [security] Address a web cache-poisoning issue reported in
|urllib.parse.parse_qsl().
|
|urllib.parse will only us "&" as query string separator by default
|instead of both ";" and "&" as allowed in earlier versions. An optional
|argument seperator with default value "&" is added to specify the
|separator.
|
|Co-authored-by: Éric Araujo 
|Co-authored-by: Ken Jin 
|Co-authored-by: Adam Goldschmidt 
|(cherry picked from commit fcbe0cb04d35189401c0c880ebfb4311e952d776)
|---
| Doc/library/cgi.rst                           |  9 ++-
| Doc/library/urllib.parse.rst                  | 23 ++++++-
| Doc/whatsnew/3.6.rst                          | 13 ++++
| Doc/whatsnew/3.7.rst                          | 13 ++++
| Lib/cgi.py                                    | 23 ++++---
| Lib/test/test_cgi.py                          | 29 ++++++--
| Lib/test/test_urlparse.py                     | 68 +++++++++++++------
| Lib/urllib/parse.py                           | 19 ++++--
| .../2021-02-14-15-59-16.bpo-42967.YApqDS.rst  |  1 +
| 9 files changed, 152 insertions(+), 46 deletions(-)
| create mode 100644 Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst
|
|diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst
|index 0b1aead9dd..f0ec7e8cc6 100644
|--- a/Doc/library/cgi.rst
|+++ b/Doc/library/cgi.rst
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
3 out of 3 hunks ignored
[...]
patching file cgi.py
patching file test/test_cgi.py
patching file test/test_urlparse.py
patching file urllib/parse.py
patching file NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst

Adjust the changes as appropriate:

$ rm -r NEWS.d/
$ git status
HEAD detached from release-pypy3.7-v7.3.3
You are in the middle of an am session.
  (fix conflicts and then run "git am --continue")
  (use "git am --skip" to skip this patch)
  (use "git am --abort" to restore the original branch)

Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git restore ..." to discard changes in working directory)
	modified:   cgi.py
	modified:   test/test_cgi.py
	modified:   test/test_urlparse.py
	modified:   urllib/parse.py

no changes added to commit (use "git add" and/or "git commit -a")
$ git add cgi.py test/test_cgi.py test/test_urlparse.py urllib/parse.py

And finally let git am commit the changes for you:

$ git am --continue
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)

PyPy is an alternative Python implementation. While it does replace a large part of the interpreter, a large part of the standard library is shared with CPython. As a result, PyPy is frequently affected by the same vulnerabilities as CPython, and we have to backport security fixes to it.

Backporting security fixes inside CPython is relatively easy. All main Python branches are in a single repository, so it’s just a matter of cherry-picking the commits. Normally, you can easily move patches between two related git repositories using git-style patches but this isn’t going to work for two repositories with unrelated histories.

Does this mean manually patching PyPy and rewriting commit messages by hand? Luckily, there’s a relatively simple git am trick that can help you avoid that.

Roughly, the idea is to:

1. Create a git-format-patch of the change to backport.

2. Attempt to apply the change via git am — it will fail and your repository will be left in middle of an am session.

3. Apply the change via patch.

4. git add the changes.

5. Finally, call git am --continue to finish, wrapping your changes in the original commit metadata.

For example, let’s try backporting CVE-2021-23336 (parameter cloaking) fix:

First, grab the relevant patch from the CPython repository:

$ git format-patch -1 d0d4d30882fe3ab9b1badbecf5d15d94326fd13e
0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch

Then, inside the local clone of a random PyPy git mirror:

$ git am -3 ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
error: sha1 information is lacking or useless (Doc/library/cgi.rst).
error: could not build fake ancestor
Patch failed at 0001 bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Now enter the directory that stdlib resides in, and apply the patch manually, skipping any missing files:

$ patch -p2 < ~/git/cpython/0001-3.7-bpo-42967-only-use-as-a-query-string-separator-G.patch
can't find file to patch at input line 39
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|From d0d4d30882fe3ab9b1badbecf5d15d94326fd13e Mon Sep 17 00:00:00 2001
|From: Senthil Kumaran 
|Date: Mon, 15 Feb 2021 10:34:14 -0800
|Subject: [PATCH] [3.7] bpo-42967: only use '&' as a query string separator
| (GH-24297)  (GH-24531)
|MIME-Version: 1.0
|Content-Type: text/plain; charset=UTF-8
|Content-Transfer-Encoding: 8bit
|
|bpo-42967: [security] Address a web cache-poisoning issue reported in
|urllib.parse.parse_qsl().
|
|urllib.parse will only us "&" as query string separator by default
|instead of both ";" and "&" as allowed in earlier versions. An optional
|argument seperator with default value "&" is added to specify the
|separator.
|
|Co-authored-by: Éric Araujo 
|Co-authored-by: Ken Jin 
|Co-authored-by: Adam Goldschmidt 
|(cherry picked from commit fcbe0cb04d35189401c0c880ebfb4311e952d776)
|---
| Doc/library/cgi.rst                           |  9 ++-
| Doc/library/urllib.parse.rst                  | 23 ++++++-
| Doc/whatsnew/3.6.rst                          | 13 ++++
| Doc/whatsnew/3.7.rst                          | 13 ++++
| Lib/cgi.py                                    | 23 ++++---
| Lib/test/test_cgi.py                          | 29 ++++++--
| Lib/test/test_urlparse.py                     | 68 +++++++++++++------
| Lib/urllib/parse.py                           | 19 ++++--
| .../2021-02-14-15-59-16.bpo-42967.YApqDS.rst  |  1 +
| 9 files changed, 152 insertions(+), 46 deletions(-)
| create mode 100644 Misc/NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst
|
|diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst
|index 0b1aead9dd..f0ec7e8cc6 100644
|--- a/Doc/library/cgi.rst
|+++ b/Doc/library/cgi.rst
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
3 out of 3 hunks ignored
[...]
patching file cgi.py
patching file test/test_cgi.py
patching file test/test_urlparse.py
patching file urllib/parse.py
patching file NEWS.d/next/Security/2021-02-14-15-59-16.bpo-42967.YApqDS.rst

Adjust the changes as appropriate:

$ rm -r NEWS.d/
$ git status
HEAD detached from release-pypy3.7-v7.3.3
You are in the middle of an am session.
  (fix conflicts and then run "git am --continue")
  (use "git am --skip" to skip this patch)
  (use "git am --abort" to restore the original branch)

Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git restore ..." to discard changes in working directory)
	modified:   cgi.py
	modified:   test/test_cgi.py
	modified:   test/test_urlparse.py
	modified:   urllib/parse.py

no changes added to commit (use "git add" and/or "git commit -a")
$ git add cgi.py test/test_cgi.py test/test_urlparse.py urllib/parse.py

And finally let git am commit the changes for you:

$ git am --continue
Applying: bpo-42967: only use '&' as a query string separator (GH-24297) (GH-24531)

February 23 2021

Why not rely on app developer to handle security?

Michał Górny (mgorny) February 23, 2021, 20:16

One of the comments to the The modern packager’s security nightmare post posed a very important question: why is it bad to depend on the app developer to address security issues? In fact, I believe it is important enough to justify a whole post discussing the problem. To clarify, the wider context is bundling dependencies, i.e. relying on the application developer to ensure that all the dependencies included with the application to be free of vulnerabilities.

In my opinion, the root of security in open source software is widely understood auditing. Since the code is public, everyone can read it, analyze it, test it. However, with a typical system install including thousands of packages from hundreds of different upstreams, it is really impossible even for large companies (not to mention individuals) to be able to audit all that code. Instead, we assume that with large enough number of eyes looking at the code, all vulnerabilities will eventually be found and published.

On top of auditing we add trust. Today, CVE authorities are at the root of our vulnerability trust. We trust them to reliably publish reports of vulnerabilities found in various packages. However, once again we can’t expect users to manually make sure that the huge number of the packages they are running are free of vulnerabilities. Instead, the trust is hierarchically moved down to software authors and distributions.

Both software authors and distribution packagers share a common goal — ensuring that their end users are running working, secure software. Why do I believe then that the user’s trust is better placed in distribution packagers than in software authors? I am going to explain this in three points.

How many entities do you trust?

The following graph depicts a fragment of dependency chain between a few C libraries and end-user programs. The right-hand side depicts leaf packages, those intended for the user. On the left side, you can see some of the libraries they use, or that their dependent libraries use. All of this stems from OpenSSL.

Now imagine that a vulnerability is found in OpenSSL. Let’s discuss what would happen in two worlds: one where the distribution is responsible for ensuring the security of all the packages, and another one where upstreams bundle dependencies and are expected to handle it.

In the distribution world, the distribution maintainers are expected to ensure that every node of that dependency graph is secure. If a vulnerability is found in OpenSSL, it is their responsibility to realize that and update the vulnerable package. The end users effectively have to trust distribution maintainers to keep their systems secure.

In the bundled dependency world, the maintainer of every successive node needs to ensure the security of its dependencies. First, the maintainers of cURL, Transmission, Tor, Synergy and Qt5 need to realize that OpenSSL has a new vulnerability, update their bundled versions and make new releases of their software. Then, the maintainers of poppler, cmake, qemu and Transmission (again) need to update their bundled versions of cURL to transitively avoid the vulnerable OpenSSL version. Same goes for the maintainers of Synergy (again) and KeepAssXC that have to update their bundled Qt5. Finally, the maintainers of Inkscape and XeTeX need to update their bundled poppler.

Even if we disregard the amount of work involved and the resulting slowdown in deploying the fix, the end user needs to trust 11 different entities to fix their respective software packages not to ship (transitively) a vulnerable version of OpenSSL. And the most absurd thing is, they will nevertheless need to trust their distribution vendor to actually ship all these updated packages.

I think that we can mostly agree that trusting a single entity provides much smaller attack surface than trusting tens or hundreds of different entities.

The above example has been placed in the C world of relatively few relatively large libraries. The standard C library is pretty fat, the standard C++ library is much fatter. A typical examples of the C approach are libraries like libgcrypt — a single library providing a lot of cryptographic primitives such as ciphers along with modes, hashes, MACs, RNGs and related functions — or OpenSSL — providing both crypto primitives and a TLS implementation on top of it. A common Rust/Cargo approach (though things are not always done this way) is to have a separate crate for every algorithm, in every variant. I’m not saying this is wrong — in fact, I actually like the idea of small libraries that do one thing well. However, it multiplies the aforementioned problem thousandfold.

The bus factor

The second part of the problem is the bus factor.

A typical ‘original’ Linux distribution involves a few hundred developers working together on software. There are much smaller distributions (or forks) but they are generally building on something bigger. These distributions generally have dedicated teams in place to handle security, as well as mechanisms to help them with their work.

For example, in Gentoo security issues can be usually tackled from two ends. On one end, there’s the package maintainer who takes care of daily maintenance tasks and usually notices vulnerability fixes through new releases. On the other end, there’s a dedicated Security team whose members are responsible for monitoring new CVEs. Most of the time, these people work together to resolve security issues, with the maintainer having specific knowledge of the package in question and the Security team sharing general security experience and monitoring the process.

Should any maintainer be away or otherwise unable to fix the vulnerability quickly, the Security team can chime in and take care of whatever needs to be done. These people are specifically focused on that one job, and this means that the chances of things going sideways are comparatively small. Even if the whole distribution were to suddenly disappear, the users have a good chance of noticing that.

Besides a few very large software packages, most of the software projects are small. It is not uncommon for a software package to be maintained by a single person. Now, how many dependencies can a single person or a small team effectively maintain? Even with the best interest at heart, it is unlikely that a software developer whose primary goal is to work on code of the particular project can be reasonably expected to share the same level of dedication and experience regarding the project’s dependencies as dedicated distribution maintainers who are doing full-time maintenance work on them.

Even if we could reasonably assume that we can trust all upstreams to do their best to ensure that their dependencies are not vulnerable, it is inevitable that some of them will not be able to handle this timely. In fact, some projects suddenly become abandoned and then vulnerabilities are not handled at all. Now, the problem is not only that it might happen. The problem is how to detect the problem early, and how to deal with it. Can you be reasonable expected to monitor hundreds of upstreams for activity? Again, the responsibility falls on distribution developers who would have to resolve these issues independently.

How much testing can you perform?

The third point is somewhat less focused on security, and more on bugs in general. Bundling dependencies not only defers handling security in packages to the application developers but also all other upgrades. The comment author argues: I want to ship software that I know works, and I can only do that if I know what my dependencies actually are so I can test against them. That’s a valid point but there’s a but: how many real life scenarios can you actually test?

Let’s start with the most basic stuff. Your CI most likely runs on a 64-bit x86 system. Some projects test more but it’s still a very limited hardware set to test. If one of your dependencies is broken on non-x86 architecture, your testing is unlikely to catch that. Even if the authors of that dependency are informed about the problem and release a fixed version, you won’t know that the upgrade is necessary unless someone reports the problem to you (and you may actually have a problem attributing the issue to the particular dependency).

In reality, things aren’t always this good. Not all upstreams release fixes quickly. Distribution packagers sometimes have to backport or even apply custom patches to make things work. If packages bundle dependencies, it is not sufficient to apply the fix at the root — it needs to be applied to all packages bundling the dependency. In the end, it is even possible that different upstreams will start applying different patches to the same dependencies to resolve the same problem independently reported to all of them. This means more maintenance work for you, and a maintenance nightmare for distributions.

There are also other kinds of issues that CIs often don’t catch. ‘Unexpected’ system setup, different locale, additional packages installed (Python is sometimes quite fragile to that). Your testing can’t really predict all possible scenarios and protect against them. Pinning to a dependency that you know to be good for you does not actually guarantee that it will be good for all your end users. By blocking upgrades, you may actually expose them to bugs that were already known and fixed.

Summary

Bundling dependencies is bad. You can reasonably assume that you’re going to do a good job at maintaining your package and keeping its bundled dependencies secure. However, you can’t guarantee that the maintainers of other packages involved will do the same. And you can’t reasonably expect the end user to place trust in the security of his system to hundreds of different people. The stakes are high and the risk is huge.

The number of entities involved is just too great. You can’t expect anyone to reasonably monitor them, and with many projects having no more than a single developer, you can’t guarantee that the fixes will be handled promptly and that the package in question will not be abandoned. A single package in middle of a dependency chain could effectively render all its reverse dependencies vulnerable, and multiply the work of their maintainers who have to locally fix the problem.

In the end, the vast majority of Linux users need to trust their distribution to ensure that the packages shipped to them are not vulnerable. While you might think that letting you handle security makes things easier for us, it doesn’t. We still need to monitor all the packages and their dependencies. The main difference is that we can fix it in one place, while upstreams have to fix it everywhere. And while they do, we need to ensure that all of them actually did that, and often it is hard to even find all the bundled dependencies (including inline code copied from other projects) as there are no widely followed standards for doing this.

So even if we ignored all the other technical downsides to bundled dependencies, the sum total of work needed to keep packages bundling it secure is much greater than the cost of unbundling the dependencies.

You should ask yourself the following question: do you really want to be responsible for all these dependencies? As a software developer, you want to focus on writing code. You want to make sure that your application works. There are other people who are ready and willing to take care of the ecosystem your software is going to run in. Fit your program into the environment, instead of building an entirely new world for it. When you do that, you’ve effectively replicating a subset of a distribution. Expecting every single application developer to do the necessary work (and to have the necessary knowledge) does not scale.

One of the comments to the The modern packager’s security nightmare post posed a very important question: why is it bad to depend on the app developer to address security issues? In fact, I believe it is important enough to justify a whole post discussing the problem. To clarify, the wider context is bundling dependencies, i.e. relying on the application developer to ensure that all the dependencies included with the application to be free of vulnerabilities.

In my opinion, the root of security in open source software is widely understood auditing. Since the code is public, everyone can read it, analyze it, test it. However, with a typical system install including thousands of packages from hundreds of different upstreams, it is really impossible even for large companies (not to mention individuals) to be able to audit all that code. Instead, we assume that with large enough number of eyes looking at the code, all vulnerabilities will eventually be found and published.

On top of auditing we add trust. Today, CVE authorities are at the root of our vulnerability trust. We trust them to reliably publish reports of vulnerabilities found in various packages. However, once again we can’t expect users to manually make sure that the huge number of the packages they are running are free of vulnerabilities. Instead, the trust is hierarchically moved down to software authors and distributions.

Both software authors and distribution packagers share a common goal — ensuring that their end users are running working, secure software. Why do I believe then that the user’s trust is better placed in distribution packagers than in software authors? I am going to explain this in three points.

How many entities do you trust?

The following graph depicts a fragment of dependency chain between a few C libraries and end-user programs. The right-hand side depicts leaf packages, those intended for the user. On the left side, you can see some of the libraries they use, or that their dependent libraries use. All of this stems from OpenSSL.

A graph depicting dependencies between a few C libraries and end-user programs

Now imagine that a vulnerability is found in OpenSSL. Let’s discuss what would happen in two worlds: one where the distribution is responsible for ensuring the security of all the packages, and another one where upstreams bundle dependencies and are expected to handle it.

In the distribution world, the distribution maintainers are expected to ensure that every node of that dependency graph is secure. If a vulnerability is found in OpenSSL, it is their responsibility to realize that and update the vulnerable package. The end users effectively have to trust distribution maintainers to keep their systems secure.

In the bundled dependency world, the maintainer of every successive node needs to ensure the security of its dependencies. First, the maintainers of cURL, Transmission, Tor, Synergy and Qt5 need to realize that OpenSSL has a new vulnerability, update their bundled versions and make new releases of their software. Then, the maintainers of poppler, cmake, qemu and Transmission (again) need to update their bundled versions of cURL to transitively avoid the vulnerable OpenSSL version. Same goes for the maintainers of Synergy (again) and KeepAssXC that have to update their bundled Qt5. Finally, the maintainers of Inkscape and XeTeX need to update their bundled poppler.

Even if we disregard the amount of work involved and the resulting slowdown in deploying the fix, the end user needs to trust 11 different entities to fix their respective software packages not to ship (transitively) a vulnerable version of OpenSSL. And the most absurd thing is, they will nevertheless need to trust their distribution vendor to actually ship all these updated packages.

I think that we can mostly agree that trusting a single entity provides much smaller attack surface than trusting tens or hundreds of different entities.

The above example has been placed in the C world of relatively few relatively large libraries. The standard C library is pretty fat, the standard C++ library is much fatter. A typical examples of the C approach are libraries like libgcrypt — a single library providing a lot of cryptographic primitives such as ciphers along with modes, hashes, MACs, RNGs and related functions — or OpenSSL — providing both crypto primitives and a TLS implementation on top of it. A common Rust/Cargo approach (though things are not always done this way) is to have a separate crate for every algorithm, in every variant. I’m not saying this is wrong — in fact, I actually like the idea of small libraries that do one thing well. However, it multiplies the aforementioned problem thousandfold.

The bus factor

The second part of the problem is the bus factor.

A typical ‘original’ Linux distribution involves a few hundred developers working together on software. There are much smaller distributions (or forks) but they are generally building on something bigger. These distributions generally have dedicated teams in place to handle security, as well as mechanisms to help them with their work.

For example, in Gentoo security issues can be usually tackled from two ends. On one end, there’s the package maintainer who takes care of daily maintenance tasks and usually notices vulnerability fixes through new releases. On the other end, there’s a dedicated Security team whose members are responsible for monitoring new CVEs. Most of the time, these people work together to resolve security issues, with the maintainer having specific knowledge of the package in question and the Security team sharing general security experience and monitoring the process.

Should any maintainer be away or otherwise unable to fix the vulnerability quickly, the Security team can chime in and take care of whatever needs to be done. These people are specifically focused on that one job, and this means that the chances of things going sideways are comparatively small. Even if the whole distribution were to suddenly disappear, the users have a good chance of noticing that.

Besides a few very large software packages, most of the software projects are small. It is not uncommon for a software package to be maintained by a single person. Now, how many dependencies can a single person or a small team effectively maintain? Even with the best interest at heart, it is unlikely that a software developer whose primary goal is to work on code of the particular project can be reasonably expected to share the same level of dedication and experience regarding the project’s dependencies as dedicated distribution maintainers who are doing full-time maintenance work on them.

Even if we could reasonably assume that we can trust all upstreams to do their best to ensure that their dependencies are not vulnerable, it is inevitable that some of them will not be able to handle this timely. In fact, some projects suddenly become abandoned and then vulnerabilities are not handled at all. Now, the problem is not only that it might happen. The problem is how to detect the problem early, and how to deal with it. Can you be reasonable expected to monitor hundreds of upstreams for activity? Again, the responsibility falls on distribution developers who would have to resolve these issues independently.

How much testing can you perform?

The third point is somewhat less focused on security, and more on bugs in general. Bundling dependencies not only defers handling security in packages to the application developers but also all other upgrades. The comment author argues: I want to ship software that I know works, and I can only do that if I know what my dependencies actually are so I can test against them. That’s a valid point but there’s a but: how many real life scenarios can you actually test?

Let’s start with the most basic stuff. Your CI most likely runs on a 64-bit x86 system. Some projects test more but it’s still a very limited hardware set to test. If one of your dependencies is broken on non-x86 architecture, your testing is unlikely to catch that. Even if the authors of that dependency are informed about the problem and release a fixed version, you won’t know that the upgrade is necessary unless someone reports the problem to you (and you may actually have a problem attributing the issue to the particular dependency).

In reality, things aren’t always this good. Not all upstreams release fixes quickly. Distribution packagers sometimes have to backport or even apply custom patches to make things work. If packages bundle dependencies, it is not sufficient to apply the fix at the root — it needs to be applied to all packages bundling the dependency. In the end, it is even possible that different upstreams will start applying different patches to the same dependencies to resolve the same problem independently reported to all of them. This means more maintenance work for you, and a maintenance nightmare for distributions.

There are also other kinds of issues that CIs often don’t catch. ‘Unexpected’ system setup, different locale, additional packages installed (Python is sometimes quite fragile to that). Your testing can’t really predict all possible scenarios and protect against them. Pinning to a dependency that you know to be good for you does not actually guarantee that it will be good for all your end users. By blocking upgrades, you may actually expose them to bugs that were already known and fixed.

Summary

Bundling dependencies is bad. You can reasonably assume that you’re going to do a good job at maintaining your package and keeping its bundled dependencies secure. However, you can’t guarantee that the maintainers of other packages involved will do the same. And you can’t reasonably expect the end user to place trust in the security of his system to hundreds of different people. The stakes are high and the risk is huge.

The number of entities involved is just too great. You can’t expect anyone to reasonably monitor them, and with many projects having no more than a single developer, you can’t guarantee that the fixes will be handled promptly and that the package in question will not be abandoned. A single package in middle of a dependency chain could effectively render all its reverse dependencies vulnerable, and multiply the work of their maintainers who have to locally fix the problem.

In the end, the vast majority of Linux users need to trust their distribution to ensure that the packages shipped to them are not vulnerable. While you might think that letting you handle security makes things easier for us, it doesn’t. We still need to monitor all the packages and their dependencies. The main difference is that we can fix it in one place, while upstreams have to fix it everywhere. And while they do, we need to ensure that all of them actually did that, and often it is hard to even find all the bundled dependencies (including inline code copied from other projects) as there are no widely followed standards for doing this.

So even if we ignored all the other technical downsides to bundled dependencies, the sum total of work needed to keep packages bundling it secure is much greater than the cost of unbundling the dependencies.

You should ask yourself the following question: do you really want to be responsible for all these dependencies? As a software developer, you want to focus on writing code. You want to make sure that your application works. There are other people who are ready and willing to take care of the ecosystem your software is going to run in. Fit your program into the environment, instead of building an entirely new world for it. When you do that, you’ve effectively replicating a subset of a distribution. Expecting every single application developer to do the necessary work (and to have the necessary knowledge) does not scale.

February 19 2021

The modern packager’s security nightmare

Michał Górny (mgorny) February 19, 2021, 22:50

One of the most important tasks of the distribution packager is to ensure that the software shipped to our users is free of security vulnerabilities. While finding and fixing the vulnerable code is usually considered upstream’s responsibility, the packager needs to ensure that all these fixes reach the end users ASAP. With the aid of central package management and dynamic linking, the Linux distributions have pretty much perfected the deployment of security fixes. Ideally, fixing a vulnerable dependency is as simple as patching a single shared library via the distribution’s automated update system.

Of course, this works only if the package in question is actually following good security practices. Over the years, many Linux distributions (at the very least, Debian, Fedora and Gentoo) have been fighting these bad practices with some success. However, today the times have changed. Today, for every 10 packages fixed, a completely new ecosystem emerges with the bad security practices at its central point. Go, Rust and to some extent Python are just a few examples of programming languages that have integrated the bad security practices into the very fabric of their existence, and recreated the same old problems in entirely new ways.

The root issue of bundling dependencies has been discussed many times before. The Gentoo Wiki explains why you should not bundle dependencies, and links to more material about it. I would like to take a bit wider approach, and discuss not only bundling (or vendoring) dependencies but also two closely relevant problems: static linking and pinning dependencies.

Static linking

In the simplest words, static linking means embedding your program’s dependencies directly into the program image. The term is generally used in contrast to dynamic linking (or dynamic loading) that keep the dependent libraries in separate files that are loaded at program’s startup (or runtime).

Why is static linking bad? The primary problem is that since they become an integral part of the program, they can not be easily replaced by another version. If it turns out that one of the libraries is vulnerable, you have to relink the whole program against the new version. This also implies that you need to have a system that keeps track of what library versions are used in individual programs.

While you might think that rebuilding a lot of packages is only a problem for source distributions, you are wrong. While indeed the users of source distributions could be impacted a lot, as their systems remain vulnerable for a long time needed to rebuild a lot of packages, a similar problem affects binary distributions. After all, the distributions need to rebuild all affected programs in order to fully ship the fix to their end users which also involves some delay.

Comparatively, shipping a new version of a shared library takes much less time and fixes all affected programs almost instantly (modulo the necessity of restarting them).

The extreme case of static linking is to distribute proprietary software that is statically linked to its dependencies. This is primarily done to ensure that the software can be run easily on a variety of systems without requiring the user to install its dependencies manually. However, this scenario is really a form of bundling dependencies, so it will be discussed in the respective section.

However, static linking has also been historically used for system programs that were meant to keep working even if their dependent libraries became broken.

In modern packages, static linking is used for another reason entirely — because they do not require the modern programming languages to have a stable ABI. The Go compiler does not need to be concerned about emitting code that would be binary compatible with the code coming from a previous version. It works around the problem by requiring you to rebuild everything every time the compiler is upgraded.

To follow the best practices, we strongly discourage static linking in C and its derivatives. However, we can’t do much about languages such as Go or Rust that put static linking at the core of their design and have time and again stated publicly that they will not switch to dynamic linking of dependencies.

Pinning dependencies

While static linking is bad, at least it provides a reasonably clear way for automatic updates (and therefore the propagation of vulnerability fixes) to happen, pinning dependencies means requiring a specific version of your program’s dependencies to be installed. While the exact results depend on the ecosystem and the exact way of pinning the dependency, generally it means that at least some users of your package will not be able to automatically update the dependencies to newer versions.

That might not seem that bad at first. However, it means that if a bug fix or — even more importantly — a vulnerability fix is released for the dependency, the users will not get it unless you update the pin and make a new release. And then, if somebody else pins your package, then that pin will also need to be updated and released. And the chain goes on. Not to mention what happens if some package just happens to indirectly pin to two different versions of the same dependency!

Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. However, all that has another underlying problem — the combination of not being concerned with API stability on upstream part, and not wishing to unnecessarily update working code (that uses deprecated API) on downstream part. Truth is, pinning makes this worse because it sweeps the problem under the carpet, and actively encourages people to develop their code against specific versions of their dependencies rather than against a stable public API. Hyrum’s Law in practice.

Dependency pinning can have really extreme consequences. Unless you make sure to update your pins often, you may one day find yourself having to take a sudden leap — because you have relied on a very old version of a dependency that is now known to be vulnerable, and in order to update it you suddenly have to rewrite a lot of code to follow the API changes. Long term, this approach simply does not scale anymore, the effort needed to keep things working grows exponentially.

We try hard to unpin the dependencies and test packages with the newest versions of them. However, often we end up discovering that the newer versions of dependencies simply are not compatible with the packages in question. Sadly, upstreams often either ignore reports of these incompatibilities or even are actively hostile to us for not following their pins.

Bundling/vendoring dependencies

Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used.

In open source software, bundling usually means either including the sources of your dependencies along with your program or making the build system fetch them automatically, and then building them along with the program. In closed source software, it usually means linking the program to its dependencies statically or including the dependency libraries along with the program.

The baseline problem is the same as with pinned dependencies — if one of them turns out to be buggy or vulnerable, the users need to wait for a new release to update the bundled dependency. In open source software or closed source software using dynamic libraries, the packager has at least a reasonable chance of replacing the problematic dependency or unbundling it entirely (i.e. forcing the system library). In statically linked closed source software, it is often impossible to even reliably determine what libraries were actually used, not to mention their exact versions. Your distribution can no longer reliably monitor security vulnerabilities; the trust is shifted to software vendors.

However, modern software sometimes takes a step further — and vendor modified dependencies. The horror of it! Now not only the packager needs to work to replace the library but often has to actually figure out what was changed compared to the original version, and rebase the changes. In worst cases, the code becomes disconnected from upstream to the point that the program author is no longer capable of updating the vendored dependency properly.

Sadly, this kind of vendoring is becoming more common with the rapid development happening these days. The cause is twofold. On one hand, downstream consumers find it easier to fork and patch a dependency than to work with upstreams. On the other hand, many upstreams are not really concerned with fixing bugs and feature requests that do not affect their own projects. Even if the fork is considered only as a stop-gap measure, it often takes a real lot of effort to push the changes upstream afterwards and re-synchronize the codebases.

We are strongly opposed to bundling dependencies. Whenever possible, we try to unbundle them — sometimes having to actually patch the build systems to reuse system libraries. However, this is a lot of work, and often it is not even possible because of custom patching, including the kind of patching that has been explicitly rejected upstream. To list a few examples — Mozilla products rely on SQLite 3 patches that collide with regular usage of this library, Rust bundles a fork of LLVM.

Summary

Static linking, dependency pinning and bundling are three bad practices that have serious impact on the time and effort needed to eliminate vulnerabilities from production systems. They can make the difference between being able to replace a vulnerable library within a few minutes and having to spend a lot of effort and time in locating multiple copies of the vulnerable library, patching and rebuilding all the software including them.

The major Linux distributions had policies against these practices for a very long time, and have been putting a lot of effort into eliminating them. Nevertheless, it feels more and more like Sisyphean task. While we have been able to successfully resolve these problems in many packages, whole new ecosystems were built on top of these bad practices — and it does not seem that upstreams care about fixing them at all.

New programming languages such as Go and Rust rely entirely on static linking, and there’s nothing we can do about it. Instead of packaging the dependencies and having programs use the newest versions, we just fetch the versions pinned by upstream and make big blobs out of it. And while upstreams brag how they magically resolved all security issues you could ever think of (entirely ignoring other classes of security issues than memory-related), we just hope that we won’t suddenly be caught with our pants down when a common pinned dependency of many packages turns out to be vulnerable.

Added 2021-04-27: Deadlock vulnerability through embedded app-emulation/containers-storage (CVE-2021-20291) is an example how a vulnerability in a single Go package requires hunting all affected ebuilds.

One of the most important tasks of the distribution packager is to ensure that the software shipped to our users is free of security vulnerabilities. While finding and fixing the vulnerable code is usually considered upstream’s responsibility, the packager needs to ensure that all these fixes reach the end users ASAP. With the aid of central package management and dynamic linking, the Linux distributions have pretty much perfected the deployment of security fixes. Ideally, fixing a vulnerable dependency is as simple as patching a single shared library via the distribution’s automated update system.

Of course, this works only if the package in question is actually following good security practices. Over the years, many Linux distributions (at the very least, Debian, Fedora and Gentoo) have been fighting these bad practices with some success. However, today the times have changed. Today, for every 10 packages fixed, a completely new ecosystem emerges with the bad security practices at its central point. Go, Rust and to some extent Python are just a few examples of programming languages that have integrated the bad security practices into the very fabric of their existence, and recreated the same old problems in entirely new ways.

The root issue of bundling dependencies has been discussed many times before. The Gentoo Wiki explains why you should not bundle dependencies, and links to more material about it. I would like to take a bit wider approach, and discuss not only bundling (or vendoring) dependencies but also two closely relevant problems: static linking and pinning dependencies.

Static linking

In the simplest words, static linking means embedding your program’s dependencies directly into the program image. The term is generally used in contrast to dynamic linking (or dynamic loading) that keep the dependent libraries in separate files that are loaded at program’s startup (or runtime).

Why is static linking bad? The primary problem is that since they become an integral part of the program, they can not be easily replaced by another version. If it turns out that one of the libraries is vulnerable, you have to relink the whole program against the new version. This also implies that you need to have a system that keeps track of what library versions are used in individual programs.

While you might think that rebuilding a lot of packages is only a problem for source distributions, you are wrong. While indeed the users of source distributions could be impacted a lot, as their systems remain vulnerable for a long time needed to rebuild a lot of packages, a similar problem affects binary distributions. After all, the distributions need to rebuild all affected programs in order to fully ship the fix to their end users which also involves some delay.

Comparatively, shipping a new version of a shared library takes much less time and fixes all affected programs almost instantly (modulo the necessity of restarting them).

The extreme case of static linking is to distribute proprietary software that is statically linked to its dependencies. This is primarily done to ensure that the software can be run easily on a variety of systems without requiring the user to install its dependencies manually. However, this scenario is really a form of bundling dependencies, so it will be discussed in the respective section.

However, static linking has also been historically used for system programs that were meant to keep working even if their dependent libraries became broken.

In modern packages, static linking is used for another reason entirely — because they do not require the modern programming languages to have a stable ABI. The Go compiler does not need to be concerned about emitting code that would be binary compatible with the code coming from a previous version. It works around the problem by requiring you to rebuild everything every time the compiler is upgraded.

To follow the best practices, we strongly discourage static linking in C and its derivatives. However, we can’t do much about languages such as Go or Rust that put static linking at the core of their design and have time and again stated publicly that they will not switch to dynamic linking of dependencies.

Pinning dependencies

While static linking is bad, at least it provides a reasonably clear way for automatic updates (and therefore the propagation of vulnerability fixes) to happen, pinning dependencies means requiring a specific version of your program’s dependencies to be installed. While the exact results depend on the ecosystem and the exact way of pinning the dependency, generally it means that at least some users of your package will not be able to automatically update the dependencies to newer versions.

That might not seem that bad at first. However, it means that if a bug fix or — even more importantly — a vulnerability fix is released for the dependency, the users will not get it unless you update the pin and make a new release. And then, if somebody else pins your package, then that pin will also need to be updated and released. And the chain goes on. Not to mention what happens if some package just happens to indirectly pin to two different versions of the same dependency!

Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. However, all that has another underlying problem — the combination of not being concerned with API stability on upstream part, and not wishing to unnecessarily update working code (that uses deprecated API) on downstream part. Truth is, pinning makes this worse because it sweeps the problem under the carpet, and actively encourages people to develop their code against specific versions of their dependencies rather than against a stable public API. Hyrum’s Law in practice.

Dependency pinning can have really extreme consequences. Unless you make sure to update your pins often, you may one day find yourself having to take a sudden leap — because you have relied on a very old version of a dependency that is now known to be vulnerable, and in order to update it you suddenly have to rewrite a lot of code to follow the API changes. Long term, this approach simply does not scale anymore, the effort needed to keep things working grows exponentially.

We try hard to unpin the dependencies and test packages with the newest versions of them. However, often we end up discovering that the newer versions of dependencies simply are not compatible with the packages in question. Sadly, upstreams often either ignore reports of these incompatibilities or even are actively hostile to us for not following their pins.

Bundling/vendoring dependencies

Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used.

In open source software, bundling usually means either including the sources of your dependencies along with your program or making the build system fetch them automatically, and then building them along with the program. In closed source software, it usually means linking the program to its dependencies statically or including the dependency libraries along with the program.

The baseline problem is the same as with pinned dependencies — if one of them turns out to be buggy or vulnerable, the users need to wait for a new release to update the bundled dependency. In open source software or closed source software using dynamic libraries, the packager has at least a reasonable chance of replacing the problematic dependency or unbundling it entirely (i.e. forcing the system library). In statically linked closed source software, it is often impossible to even reliably determine what libraries were actually used, not to mention their exact versions. Your distribution can no longer reliably monitor security vulnerabilities; the trust is shifted to software vendors.

However, modern software sometimes takes a step further — and vendor modified dependencies. The horror of it! Now not only the packager needs to work to replace the library but often has to actually figure out what was changed compared to the original version, and rebase the changes. In worst cases, the code becomes disconnected from upstream to the point that the program author is no longer capable of updating the vendored dependency properly.

Sadly, this kind of vendoring is becoming more common with the rapid development happening these days. The cause is twofold. On one hand, downstream consumers find it easier to fork and patch a dependency than to work with upstreams. On the other hand, many upstreams are not really concerned with fixing bugs and feature requests that do not affect their own projects. Even if the fork is considered only as a stop-gap measure, it often takes a real lot of effort to push the changes upstream afterwards and re-synchronize the codebases.

We are strongly opposed to bundling dependencies. Whenever possible, we try to unbundle them — sometimes having to actually patch the build systems to reuse system libraries. However, this is a lot of work, and often it is not even possible because of custom patching, including the kind of patching that has been explicitly rejected upstream. To list a few examples — Mozilla products rely on SQLite 3 patches that collide with regular usage of this library, Rust bundles a fork of LLVM.

Summary

Static linking, dependency pinning and bundling are three bad practices that have serious impact on the time and effort needed to eliminate vulnerabilities from production systems. They can make the difference between being able to replace a vulnerable library within a few minutes and having to spend a lot of effort and time in locating multiple copies of the vulnerable library, patching and rebuilding all the software including them.

The major Linux distributions had policies against these practices for a very long time, and have been putting a lot of effort into eliminating them. Nevertheless, it feels more and more like Sisyphean task. While we have been able to successfully resolve these problems in many packages, whole new ecosystems were built on top of these bad practices — and it does not seem that upstreams care about fixing them at all.

New programming languages such as Go and Rust rely entirely on static linking, and there’s nothing we can do about it. Instead of packaging the dependencies and having programs use the newest versions, we just fetch the versions pinned by upstream and make big blobs out of it. And while upstreams brag how they magically resolved all security issues you could ever think of (entirely ignoring other classes of security issues than memory-related), we just hope that we won’t suddenly be caught with our pants down when a common pinned dependency of many packages turns out to be vulnerable.

Added 2021-04-27: Deadlock vulnerability through embedded app-emulation/containers-storage (CVE-2021-20291) is an example how a vulnerability in a single Go package requires hunting all affected ebuilds.

February 12 2021

In Memory of Kent “kentnl” Fredric

Gentoo News (GentooNews) February 12, 2021, 6:00

Gentoo mourns the sudden loss of Kent Fredric, also known to us by his IRC handle kent\n. He passed away following a tragic accident a few days ago.

Kent was an active member of the Gentoo community for many years. He tirelessly managed Gentoo’s Perl support, and was active in the Rust project as well as in many other corners. We all remember him as an enthusiastic, bright person, with lots of eye for detail and constant willingness to help out and improve things. On behalf of the world-wide Gentoo community, our heartfelt condolences go out to his family and friends.

Please join us in remembering Kent on the Gentoo forums.

Gentoo mourns the sudden loss of Kent Fredric, also known to us by his IRC handle kent\n. He passed away following a tragic accident a few days ago.

Kent was an active member of the Gentoo community for many years. He tirelessly managed Gentoo’s Perl support, and was active in the Rust project as well as in many other corners. We all remember him as an enthusiastic, bright person, with lots of eye for detail and constant willingness to help out and improve things. On behalf of the world-wide Gentoo community, our heartfelt condolences go out to his family and friends.

Please join us in remembering Kent on the Gentoo forums.

February 10 2021

lzip decompression support for xz-utils

Michał Górny (mgorny) February 10, 2021, 14:29

As of today, the most common implementation of the LZMA algorithm on open source operating systems is the xz format. However, there are a few others available. Notably, a few packages found in the Gentoo repository use the superior lzip format. Does this mean you may end up having to have separate decompressors for both formats installed? Not necessarily.

Back in 2017, I’ve entertained a curious idea. Since both lzip and xz are both container formats built on top of the original LZMA algorithm, and xz features backwards-compatible support for the earlier container format used by lzma-utils, how hard would it be to implement a decoder for the lzip format as well? Not very hard, it turned out. After all, most of the code was already there — I’ve just had to implement the additional container format. With some kind help of XZ upstream, I’ve done that.

Sadly, the patches have not been merged yet. More than three years have passed now in waiting. Today I’ve rebased them and updated to follow changes in XZ itself. For anyone interested, it can be found on the lzip2 branch of my xz fork. After building xz with my patches, it now happily decompresses .lz files in addition to the regular set. Thanks to a tiny patchset you don’t have to build yet another program to unpack a few distfiles.

As of today, the most common implementation of the LZMA algorithm on open source operating systems is the xz format. However, there are a few others available. Notably, a few packages found in the Gentoo repository use the superior lzip format. Does this mean you may end up having to have separate decompressors for both formats installed? Not necessarily.

Back in 2017, I’ve entertained a curious idea. Since both lzip and xz are both container formats built on top of the original LZMA algorithm, and xz features backwards-compatible support for the earlier container format used by lzma-utils, how hard would it be to implement a decoder for the lzip format as well? Not very hard, it turned out. After all, most of the code was already there — I’ve just had to implement the additional container format. With some kind help of XZ upstream, I’ve done that.

Sadly, the patches have not been merged yet. More than three years have passed now in waiting. Today I’ve rebased them and updated to follow changes in XZ itself. For anyone interested, it can be found on the lzip2 branch of my xz fork. After building xz with my patches, it now happily decompresses .lz files in addition to the regular set. Thanks to a tiny patchset you don’t have to build yet another program to unpack a few distfiles.

January 31 2021

New Gentoo riscv (and arm) stages

Andreas K. Hüttel (dilfridge) January 31, 2021, 21:31

♦With the help of our infrastructure team, I've finally managed to integrate the riscv stage builds with our signing and mirroring system. So now we have a riscv tab on the installation media download page, and the mirrors carry weekly signed stage3 archives for riscv64-lp64d and riscv64-lp64, in both openrc and systemd variants.

Using the same build infrastructure based on qemu, there are now also slowly updated stages for all arm variants coming to the mirrors. Please test them, and if anything does not work as expected, file bugs! The qemu-based builds are here a temporary measure; Matt Turner (mattst88) is preparing a fast multi-core arm64 machine, where this task will move to soon.

With the help of our infrastructure team, I've finally managed to integrate the riscv stage builds with our signing and mirroring system. So now we have a riscv tab on the installation media download page, and the mirrors carry weekly signed stage3 archives for riscv64-lp64d and riscv64-lp64, in both openrc and systemd variants.

Using the same build infrastructure based on qemu, there are now also slowly updated stages for all arm variants coming to the mirrors. Please test them, and if anything does not work as expected, file bugs! The qemu-based builds are here a temporary measure; Matt Turner (mattst88) is preparing a fast multi-core arm64 machine, where this task will move to soon.

January 16 2021

Distribution Kernels: module rebuilds, better ZFS support and UEFI executables

Michał Górny (mgorny) January 16, 2021, 10:12

The primary goal of the Distribution Kernel project is provide a seamless kernel upgrade experience to Gentoo users. Initially, this meant configuring, building and installing the kernel during the @world upgrade. However, you had to manually rebuild the installed kernel modules (and @module-rebuild is still broken), and sometimes additionally rebuild the initramfs after doing that.

To address this, we have introduced a new dist-kernel USE flag. This flag is automatically added to all ebuilds installing kernel modules. When it is enabled, the linux-mod eclass adds a dependency on virtual/dist-kernel package. This virtual, in turn, is bound to the newest version of dist-kernel installed. As a result, whenever you upgrade your dist-kernel all the module packages will also be rebuilt via slot rebuilds. The manual @module-rebuild should no longer be necessary!

ZFS users have pointed out that after rebuilding sys-fs/zfs-kmod package, they need to rebuild the initramfs for Dracut to include the new module. We have combined the dist-kernel rebuild feature with pkg_postinst() to rebuild the initramfs whenever zfs-kmod is being rebuilt (and the dist-kernel is used). As a result, ZFS should no longer require any manual attention — as long as rebuilds succeed, the new kernel and initramfs should be capable of running on ZFS root once the @world upgrade finishes.

Finally, we have been asked to provide support for uefi=yes Dracut option. When this option is enabled, Dracut combines the EFI stub, kernel and generated initramfs into a single UEFI executable that can be booted directly. The dist-kernels now detect this scenario, and install the generated executable in place of the kernel, so everything works as expected. Note that due to implementation limitations, we also install an empty initramfs as otherwise kernel-install.d scripts would insist on creating another initramfs. Also note that until Dracut is fixed to use correct EFI stub path, you have to set the path manually in /etc/dracut.conf:

uefi_stub=/usr/lib/systemd/boot/efi/linuxx64.efi.stub

The primary goal of the Distribution Kernel project is provide a seamless kernel upgrade experience to Gentoo users. Initially, this meant configuring, building and installing the kernel during the @world upgrade. However, you had to manually rebuild the installed kernel modules (and @module-rebuild is still broken), and sometimes additionally rebuild the initramfs after doing that.

To address this, we have introduced a new dist-kernel USE flag. This flag is automatically added to all ebuilds installing kernel modules. When it is enabled, the linux-mod eclass adds a dependency on virtual/dist-kernel package. This virtual, in turn, is bound to the newest version of dist-kernel installed. As a result, whenever you upgrade your dist-kernel all the module packages will also be rebuilt via slot rebuilds. The manual @module-rebuild should no longer be necessary!

ZFS users have pointed out that after rebuilding sys-fs/zfs-kmod package, they need to rebuild the initramfs for Dracut to include the new module. We have combined the dist-kernel rebuild feature with pkg_postinst() to rebuild the initramfs whenever zfs-kmod is being rebuilt (and the dist-kernel is used). As a result, ZFS should no longer require any manual attention — as long as rebuilds succeed, the new kernel and initramfs should be capable of running on ZFS root once the @world upgrade finishes.

Finally, we have been asked to provide support for uefi=yes Dracut option. When this option is enabled, Dracut combines the EFI stub, kernel and generated initramfs into a single UEFI executable that can be booted directly. The dist-kernels now detect this scenario, and install the generated executable in place of the kernel, so everything works as expected. Note that due to implementation limitations, we also install an empty initramfs as otherwise kernel-install.d scripts would insist on creating another initramfs. Also note that until Dracut is fixed to use correct EFI stub path, you have to set the path manually in /etc/dracut.conf:

uefi_stub=/usr/lib/systemd/boot/efi/linuxx64.efi.stub

January 15 2021

2020 in retrospect & happy new year 2021!

Gentoo News (GentooNews) January 15, 2021, 6:00

♦ Happy New Year 2021! Due to the COVID pandemic, 2020 was a year unlike any other, and this has also impacted many open source projects. Nevertheless, at Gentoo we have made some great strides forward. While we now start into 2021 with fresh energy (and maybe soon antibodies), let’s also take a look back. We’re happy to share with our community the most exciting news of the past 12 months – including numbers on Gentoo activity, our new developers, and featured changes and improvements!

Gentoo in numbers

2020 has featured a major increase in commits to the ::gentoo repository, and especially commits from non-developers. The overall number of commits has grown from 73400 to 104500 (by 42%), while the number of commits made by non-developers has grown from 5700 (8% of total) to 11000 (10.5% of total). The latter group has featured 333 unique authors in 2019, and 391 in 2020.

The ::guru repository has thrived in 2020. While 2019 left it with merely 7 contributors and a total of 86 commits, 2020 has featured 55 different contributors and 2725 commits. GURU is a user-curated repository with a trusted user model. Come join us!

There was also a major increase in Bugzilla activity. 2020 featured almost 25500 bugs reported, compared to 15000 in 2019. This is probably largely thanks to Agostino Sarubbo’s new tinderboxing effort. The total number of bugs closed in 2020 was 23500, compared to 15000 in 2019.

New developers

We’ve finished 2020 with three significant additions to the Gentoo family (in chronological order):

  1. Max Magorsch (arzano)

    ♦ Max joined us in February to help out with Gentoo Infrastructure. Since then, he already did tons of work. Just to list a few things, he has redesigned and modernized the Gentoo websites and rewritten packages.gentoo.org into the super cool form we have today.

  2. Sam James (sam)

    ♦ Sam joined us in July, and has contributed to a lot of different projects since. He is known as an active member of the Security team and multiple arch teams, as well as someone who fixes lots of bugs in different packages.

  3. Stephan Hartmann (sultan)

    ♦ Stephan joined us in September, and immediately started working on our Chromium-related packages. He has pushed commits to upstream Chromium; hopefully he’ll deal with all the specific problems that come up in Gentoo here. Thanks to him we also have finally caught up with Windows, offering our users a packaged version of Microsoft Edge.

Featured changes

The following major changes and improvements have happened in 2020:

Packages
  • ♦ Distribution Kernels: Gentoo now supports building and installing kernels entirely via the package manager. The new kernel packages also come with an (optional) stock configuration based on well-tested Fedora kernels, to ease the entry barrier and maintenance effort of Gentoo systems.

  • ♦ Wayland: Wayland support in Gentoo has progressed greatly, making it possible to run an Xorg-free desktop. Wayland is supported with large desktop environments such as KDE Plasma and GNOME, as well as with lightweight alternatives such as Sway and Wayfire. The latter makes it also possible to use Wayland to a large extent without resorting to XWayland.

  • ♦ Lua: A new framework has been created that permits multiple versions of Lua to be installed side-by-side. The vast majority of ~arch packages have already been migrated to this framework. This way, we have finally been able to unmask new (slotted!) Lua versions.

  • ♦ Python: We have managed to almost withdraw Python 2.7 from Gentoo, and upgrade the default to Python 3.8. Python 2.7 is still available as a build-time dependency for a few packages. We have additionally patched all the vulnerabilities known from later versions of Python.

Architectures
  • ARM64: ARM64 (AArch64) support has been elevated to stable status and is no longer experimental. The ARM64 project now provides automatically generated stage3 files, and is usually one of the fastest arch teams to test packages. We have worked to bring more packages to ARM64 and make it more feasible to run a full desktop!

  • PPC64: KDE Plasma is now available on PPC64, thanks to extensive testing and keywording efforts by Georgy Yakovlev.

  • RISC-V: Work on RISC-V support has started, with particular focus on the riscv64 architecture. The RISC-V project provides stage3 files and stable profiles for the soft-float (rv64imac/lp64) and hard-float (rv64gc/lp64d) ABIs, in both systemd and OpenRC variants. The arch team has managed to run Xorg already!

  • Prefix: Gentoo Prefix is once again capable of bootstrapping on the latest macOS releases, and work is underway to modernise prefix-specific ebuilds and merge them back into the main tree - this way ensuring that users get the latest software and that maintenance burden is reduced.

  • ♦ Android: The Gentoo Android project has released a new 64bit Android prefix tarball, featuring gcc-10.1.0, binutils-2.34 and glibc-2.31 in your pocket!

Infrastructure
  • ♦ packages.gentoo.org: The packages website has received many improvements towards being a central source of information on Gentoo packages. It now shows the results of QA checks, bugs, pull requests referencing a package, and a maintainer dashboard indicating stabilization candidates and outdated versions (according to Repology). Additionally, the display can be configured for your personal preferences!

  • ♦ Bugzilla: The Infrastructure team has implemented a major improvement to Gentoo Bugzilla performance. The database has been migrated to a newer database cluster, and the backend has been switched to mod_perl.

  • CI / Tinderbox: A second active tinderboxing (build testing) effort has been started, resulting in more bugs being detected and fixed early. This also includes running a variety of QA checks, as well as minimal environment builds that are helpful in detecting missing dependencies.

Other news
  • ♦ HPC adoption: The Prefix Project has published a conference proceeding on a case study of Gentoo in high energy physics. Gentoo also sees wider adoption in the HPC community such as Compute Canada and EESSI.
Discontinued projects

While Gentoo would like to support as much as our users wish for, we could not manage to continue all of the projects we’ve started in the past. With limited resources, we had to divert our time and effort from projects showing little promise and activity. The most important projects discontinued in 2020 were:

  • Architectures: Alpha and IA64 keywords were reduced to ~arch (i.e. unstable/testing only). HPPA stable keywords were limited to the most important packages only. SH (SuperH) was removed entirely. With very small number of users of these architectures, our arch teams decided that the effort in maintaining them is too great. In case of SuperH, our last available hardware died.

  • LibreSSL: By the end of 2020, we have decided to discontinue support for LibreSSL. With little to no support from various upstream projects, the effort necessary to maintain package compatibility exceeded the gain, especially given that OpenSSL has made a lot of progress since the forking point.

Thank you!

We can here describe only a few major items, and these cover by far not all that is going on. We would like to thank all Gentoo developers for their relentless everyday Gentoo work. While they are often not recognized for this work, Gentoo could not exist without them. Cheers, and let’s make 2021 even more productive!

Gentoo Fireworks Happy New Year 2021! Due to the COVID pandemic, 2020 was a year unlike any other, and this has also impacted many open source projects. Nevertheless, at Gentoo we have made some great strides forward. While we now start into 2021 with fresh energy (and maybe soon antibodies), let’s also take a look back. We’re happy to share with our community the most exciting news of the past 12 months – including numbers on Gentoo activity, our new developers, and featured changes and improvements!

Gentoo in numbers

2020 has featured a major increase in commits to the ::gentoo repository, and especially commits from non-developers. The overall number of commits has grown from 73400 to 104500 (by 42%), while the number of commits made by non-developers has grown from 5700 (8% of total) to 11000 (10.5% of total). The latter group has featured 333 unique authors in 2019, and 391 in 2020.

The ::guru repository has thrived in 2020. While 2019 left it with merely 7 contributors and a total of 86 commits, 2020 has featured 55 different contributors and 2725 commits. GURU is a user-curated repository with a trusted user model. Come join us!

There was also a major increase in Bugzilla activity. 2020 featured almost 25500 bugs reported, compared to 15000 in 2019. This is probably largely thanks to Agostino Sarubbo’s new tinderboxing effort. The total number of bugs closed in 2020 was 23500, compared to 15000 in 2019.

New developers

We’ve finished 2020 with three significant additions to the Gentoo family (in chronological order):

  1. Max Magorsch (arzano)

    Max joined us in February to help out with Gentoo Infrastructure. Since then, he already did tons of work. Just to list a few things, he has redesigned and modernized the Gentoo websites and rewritten packages.gentoo.org into the super cool form we have today.

  2. Sam James (sam)

    Sam joined us in July, and has contributed to a lot of different projects since. He is known as an active member of the Security team and multiple arch teams, as well as someone who fixes lots of bugs in different packages.

  3. Stephan Hartmann (sultan)

    Stephan joined us in September, and immediately started working on our Chromium-related packages. He has pushed commits to upstream Chromium; hopefully he’ll deal with all the specific problems that come up in Gentoo here. Thanks to him we also have finally caught up with Windows, offering our users a packaged version of Microsoft Edge.

The following major changes and improvements have happened in 2020:

Packages

  • Distribution Kernels: Gentoo now supports building and installing kernels entirely via the package manager. The new kernel packages also come with an (optional) stock configuration based on well-tested Fedora kernels, to ease the entry barrier and maintenance effort of Gentoo systems.

  • Wayland: Wayland support in Gentoo has progressed greatly, making it possible to run an Xorg-free desktop. Wayland is supported with large desktop environments such as KDE Plasma and GNOME, as well as with lightweight alternatives such as Sway and Wayfire. The latter makes it also possible to use Wayland to a large extent without resorting to XWayland.

  • Lua: A new framework has been created that permits multiple versions of Lua to be installed side-by-side. The vast majority of ~arch packages have already been migrated to this framework. This way, we have finally been able to unmask new (slotted!) Lua versions.

  • Python: We have managed to almost withdraw Python 2.7 from Gentoo, and upgrade the default to Python 3.8. Python 2.7 is still available as a build-time dependency for a few packages. We have additionally patched all the vulnerabilities known from later versions of Python.

Architectures

  • ARM64: ARM64 (AArch64) support has been elevated to stable status and is no longer experimental. The ARM64 project now provides automatically generated stage3 files, and is usually one of the fastest arch teams to test packages. We have worked to bring more packages to ARM64 and make it more feasible to run a full desktop!

  • PPC64: KDE Plasma is now available on PPC64, thanks to extensive testing and keywording efforts by Georgy Yakovlev.

  • RISC-V: Work on RISC-V support has started, with particular focus on the riscv64 architecture. The RISC-V project provides stage3 files and stable profiles for the soft-float (rv64imac/lp64) and hard-float (rv64gc/lp64d) ABIs, in both systemd and OpenRC variants. The arch team has managed to run Xorg already!

  • Prefix: Gentoo Prefix is once again capable of bootstrapping on the latest macOS releases, and work is underway to modernise prefix-specific ebuilds and merge them back into the main tree - this way ensuring that users get the latest software and that maintenance burden is reduced.

  • Android: The Gentoo Android project has released a new 64bit Android prefix tarball, featuring gcc-10.1.0, binutils-2.34 and glibc-2.31 in your pocket!

Infrastructure

  • packages.gentoo.org: The packages website has received many improvements towards being a central source of information on Gentoo packages. It now shows the results of QA checks, bugs, pull requests referencing a package, and a maintainer dashboard indicating stabilization candidates and outdated versions (according to Repology). Additionally, the display can be configured for your personal preferences!

  • Bugzilla: The Infrastructure team has implemented a major improvement to Gentoo Bugzilla performance. The database has been migrated to a newer database cluster, and the backend has been switched to mod_perl.

  • CI / Tinderbox: A second active tinderboxing (build testing) effort has been started, resulting in more bugs being detected and fixed early. This also includes running a variety of QA checks, as well as minimal environment builds that are helpful in detecting missing dependencies.

Other news

Discontinued projects

While Gentoo would like to support as much as our users wish for, we could not manage to continue all of the projects we’ve started in the past. With limited resources, we had to divert our time and effort from projects showing little promise and activity. The most important projects discontinued in 2020 were:

  • Architectures: Alpha and IA64 keywords were reduced to ~arch (i.e. unstable/testing only). HPPA stable keywords were limited to the most important packages only. SH (SuperH) was removed entirely. With very small number of users of these architectures, our arch teams decided that the effort in maintaining them is too great. In case of SuperH, our last available hardware died.

  • LibreSSL: By the end of 2020, we have decided to discontinue support for LibreSSL. With little to no support from various upstream projects, the effort necessary to maintain package compatibility exceeded the gain, especially given that OpenSSL has made a lot of progress since the forking point.

Thank you!

We can here describe only a few major items, and these cover by far not all that is going on. We would like to thank all Gentoo developers for their relentless everyday Gentoo work. While they are often not recognized for this work, Gentoo could not exist without them. Cheers, and let’s make 2021 even more productive!

December 29 2020

OpenSSL, LibreSSL, LibreTLS and all the terminological irony

Michał Górny (mgorny) December 29, 2020, 18:11

While we’re discussing the fate of LibreSSL, it’s worth noting how confusing the names of these packages became. I’d like to take this opportunity to provide a short note on what’s what.

First of all, SSL and its successor TLS are protocols used to implement network connection security. For historical reasons, many libraries carry ‘SSL’ in their name (OpenSSL, LibreSSL, PolarSSL) but nowadays they all support TLS.

OpenSSL is the ‘original’ crypto/SSL/TLS library. It is maintained independently of a specific operating system. It provides two main libraries: libcrypto and libssl (that also implements TLS).

LibreSSL is a fork of OpenSSL. It is maintained by OpenBSD as part of its base system. However, the upstream also maintains LibreSSL-portable repository that provides build system and portability glue for using it on other systems. LibreSSL provides partially compatible versions of libcrypto and libssl, and a new libtls library. Both libssl and libtls can be used for TLS support in your applications.

LibreTLS is a lightweight fork of libtls from LibreSSL that builds it against OpenSSL. This makes it possible to build programs written for libtls against OpenSSL+LibreTLS instead of LibreSSL.

So, to summarize. OpenSSL is the original, while LibreSSL is the OpenBSD fork. libtls is the LibreSSL original library, while LibreTLS is its fork for OpenSSL. Makes sense, right? And finally, despite the name, they all implement TLS.

While we’re discussing the fate of LibreSSL, it’s worth noting how confusing the names of these packages became. I’d like to take this opportunity to provide a short note on what’s what.

First of all, SSL and its successor TLS are protocols used to implement network connection security. For historical reasons, many libraries carry ‘SSL’ in their name (OpenSSL, LibreSSL, PolarSSL) but nowadays they all support TLS.

OpenSSL is the ‘original’ crypto/SSL/TLS library. It is maintained independently of a specific operating system. It provides two main libraries: libcrypto and libssl (that also implements TLS).

LibreSSL is a fork of OpenSSL. It is maintained by OpenBSD as part of its base system. However, the upstream also maintains LibreSSL-portable repository that provides build system and portability glue for using it on other systems. LibreSSL provides partially compatible versions of libcrypto and libssl, and a new libtls library. Both libssl and libtls can be used for TLS support in your applications.

LibreTLS is a lightweight fork of libtls from LibreSSL that builds it against OpenSSL. This makes it possible to build programs written for libtls against OpenSSL+LibreTLS instead of LibreSSL.

So, to summarize. OpenSSL is the original, while LibreSSL is the OpenBSD fork. libtls is the LibreSSL original library, while LibreTLS is its fork for OpenSSL. Makes sense, right? And finally, despite the name, they all implement TLS.

November 06 2020

Renaming and reshaping Scylla tables using scylla-migrator

Alexys Jacob (ultrabug) November 06, 2020, 20:11

We have recently faced a problem where some of the first Scylla tables we created on our main production cluster were not in line any more with the evolved schemas that recent tables are using.

This typical engineering problem requires either to keep those legacy tables and data queries or to migrate it to the more optimal model with the bandwagon of applications to be modified to query the data the new way… That’s something nobody likes doing but hey, we don’t like legacy at Numberly so let’s kill that one!

To overcome this challenge we used the scylla-migrator project and I thought it could be useful to share this experience.

How and why our schema evolved

When we first approached ID matching tables we chose to answer two problems at the same time: query the most recent data and keep the history of the changes per source ID.

This means that those tables included a date as part of their PRIMARY KEY while the partition key was obviously the matching table ID we wanted to lookup from:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
date timestamp,
PRIMARY KEY ((partnerid), date, id)
)
WITH CLUSTERING ORDER BY (date DESC)

Making a table with an ever changing date in the clustering key creates what we call a history table. In the schema above the uniqueness of a row is not only defined by a partner_id / id couple but also by its date!

Quick caveat: you have to be careful about the actual date timestamp resolution since you may not want to create a row for every second of the same partner_id / id couple (we use an hour resolution).

History tables are good for analytics and we also figured we could use them for batch and real time queries where we would be interested in the “most recent ids for the given partner_id” (sometimes flavored with a LIMIT):

SELECT id FROM ids_by_partnerid WHERE partner_id = "AXZAZLKDJ" ORDER BY date DESC;

As time passed, real time Kafka pipelines started to query these tables hard and were mostly interested in “all the ids known for the given partner_id“.

A sort of DISTINCT(id) is out of the scope of our table! For this we need a table schema that represents a condensed view of the data. We call them compact tables and the only difference with the history table is that the date timestamp is simply not part of the PRIMARY KEY:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
seen_date timestamp,
PRIMARY KEY ((partnerid), id)
)

To make that transition happen we thus wanted to:

  • rename history tables with an _history suffix so that they are clearly identified as such
  • get a compacted version of the tables (by keeping their old name) while renaming the date column name to seen_date
  • do it as fast as possible since we will need to stop our feeding pipeline and most of our applications during the process…

STOP: it’s not possible to rename a table in CQL!

♦ Scylla-migrator to the rescue

We decided to abuse the scylla-migrator to perform this perilous migration.

As it was originally designed to help users migrate from Cassandra to Scylla by leveraging Spark it seemed like a good fit for the task since we happen to own Spark clusters powered by Hadoop YARN.

Building scylla-migrator for Spark < 2.4

Recent scylla-migrator does not support older Spark versions. The trick is to look at the README.md git log and checkout the hopefully right commit that supports your Spark cluster version.

In our case for Spark 2.3 we used git commit bc82a57e4134452f19a11cd127bd4c6a25f75020.

On Gentoo, make sure to use dev-java/sbt-bin since the non binary version is vastly out of date and won’t build the project. You need at least version 1.3.

The scylla-migrator plan

The documentation explains that we need a config file that points to a source cluster+table and a destination cluster+table as long as they have the same schema structure…

Renaming is then as simple as duplicating the schema using CQLSH and running the migrator!

But what about our compacted version of our original table? The schema is different from the source table!…

Good news is that as long as all your columns remain present, you can also change the PRIMARY KEY of your destination table and it will still work!

This make the scylla-migrator an amazing tool to reshape or pivot tables!

  • the column date is renamed to seen_date: that’s okay, scylla-migrator supports column renaming (it’s a Spark dataframe after all)!
  • the PRIMARY KEY is different in the compacted table since we removed the ‘date‘ from the clustering columns: we’ll get a compacted table for free!
Using scylla-migrator

The documentation is a bit poor on how to submit your application to a Hadoop YARN cluster but that’s kind of expected.

It also did not mention how to connect to a SSL enabled cluster (are there people really not using SSL on the wire in their production environment?)… anyway let’s not start a flame war ♦

The trick that will save you is to know that you can append all the usual Spark options that are available in the spark-cassandra-connector!

Submitting to a Kerberos protected Hadoop YARN cluster targeting a SSL enabled Scylla cluster then looks like this:

export JAR_NAME=target/scala-2.11/scylla-migrator-assembly-0.0.1.jar
export KRB_PRINCIPAL=USERNAME

spark2-submit \
 --name ScyllaMigratorApplication \
 --class com.scylladb.migrator.Migrator  \
 --conf spark.cassandra.connection.ssl.clientAuth.enabled=True  \
 --conf spark.cassandra.connection.ssl.enabled=True  \
 --conf spark.cassandra.connection.ssl.trustStore.path=jssecacerts  \
 --conf spark.cassandra.connection.ssl.trustStore.password=JKS_PASSWORD  \
 --conf spark.cassandra.input.consistency.level=LOCAL_QUORUM \
 --conf spark.cassandra.output.consistency.level=LOCAL_QUORUM \
 --conf spark.scylla.config=config.yaml \
 --conf spark.yarn.executor.memoryOverhead=1g \
 --conf spark.blacklist.enabled=true  \
 --conf spark.blacklist.task.maxTaskAttemptsPerExecutor=1  \
 --conf spark.blacklist.task.maxTaskAttemptsPerNode=1  \
 --conf spark.blacklist.stage.maxFailedTasksPerExecutor=1  \
 --conf spark.blacklist.stage.maxFailedExecutorsPerNode=1  \
 --conf spark.executor.cores=16 \
 --deploy-mode client \
 --files jssecacerts \
 --jars ${JAR_NAME}  \
 --keytab ${KRB_PRINCIPAL}.keytab  \
 --master yarn \
 --principal ${KRB_PRINCIPAL}  \
 ${JAR_NAME}

Note that we chose to apply a higher consistency level to our reads using a LOCAL_QUORUM instead of the default LOCAL_ONE. I strongly encourage you to do the same since it’s appropriate when you’re using this kind of tool!

Column renaming is simply expressed in the configuration file like this:

# Column renaming configuration.
renames:
  - from: date
    to: seen_date
Tuning scylla-migrator

While easy to use, tuning scylla-migrator to operate those migrations as fast as possible turned out to be a real challenge (remember we have some production applications shut down during the process).

Even using 300+ Spark executors I couldn’t get my Scylla cluster utilization to more than 50% and migrating a single table with a bit more than 1B rows took almost 2 hours…

We found the best knobs to play with thanks to the help of Lubos Kosco and this blog post from ScyllaDB:

  • Increase the splitCount setting: more splits means more Spark executors will be spawned and more tasks out of it. While it might be magic on a pure Spark deployment it’s not that amazing on a Hadoop YARN one where executors are scheduled in containers with 1 core by default. We simply moved it from 256 to 384.
  • Disable compaction on destination tables schemas. This gave us a big boost and saved the day since it avoids adding the overhead of compacting while you’re pushing down data hard!

To disable compaction on a table simply:

ALTER TABLE ids_by_partnerid_history WITH compaction = {'class': 'NullCompactionStrategy'};

Remember to run a manual compaction (nodetool compact <keyspace> <table>) and to enable compaction back on your tables once you’re done!

Happy Scylla tables mangling!

We have recently faced a problem where some of the first Scylla tables we created on our main production cluster were not in line any more with the evolved schemas that recent tables are using.

This typical engineering problem requires either to keep those legacy tables and data queries or to migrate it to the more optimal model with the bandwagon of applications to be modified to query the data the new way… That’s something nobody likes doing but hey, we don’t like legacy at Numberly so let’s kill that one!

To overcome this challenge we used the scylla-migrator project and I thought it could be useful to share this experience.

How and why our schema evolved

When we first approached ID matching tables we chose to answer two problems at the same time: query the most recent data and keep the history of the changes per source ID.

This means that those tables included a date as part of their PRIMARY KEY while the partition key was obviously the matching table ID we wanted to lookup from:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
date timestamp,
PRIMARY KEY ((partnerid), date, id)
)
WITH CLUSTERING ORDER BY (date DESC)

Making a table with an ever changing date in the clustering key creates what we call a history table. In the schema above the uniqueness of a row is not only defined by a partner_id / id couple but also by its date!

Quick caveat: you have to be careful about the actual date timestamp resolution since you may not want to create a row for every second of the same partner_id / id couple (we use an hour resolution).

History tables are good for analytics and we also figured we could use them for batch and real time queries where we would be interested in the “most recent ids for the given partner_id” (sometimes flavored with a LIMIT):

SELECT id FROM ids_by_partnerid WHERE partner_id = "AXZAZLKDJ" ORDER BY date DESC;

As time passed, real time Kafka pipelines started to query these tables hard and were mostly interested in “all the ids known for the given partner_id“.

A sort of DISTINCT(id) is out of the scope of our table! For this we need a table schema that represents a condensed view of the data. We call them compact tables and the only difference with the history table is that the date timestamp is simply not part of the PRIMARY KEY:

CREATE TABLE IF NOT EXISTS ids_by_partnerid(
partnerid text,
id text,
seen_date timestamp,
PRIMARY KEY ((partnerid), id)
)

To make that transition happen we thus wanted to:

  • rename history tables with an _history suffix so that they are clearly identified as such
  • get a compacted version of the tables (by keeping their old name) while renaming the date column name to seen_date
  • do it as fast as possible since we will need to stop our feeding pipeline and most of our applications during the process…

STOP: it’s not possible to rename a table in CQL!

Scylla-migrator to the rescue

We decided to abuse the scylla-migrator to perform this perilous migration.

As it was originally designed to help users migrate from Cassandra to Scylla by leveraging Spark it seemed like a good fit for the task since we happen to own Spark clusters powered by Hadoop YARN.

Building scylla-migrator for Spark < 2.4

Recent scylla-migrator does not support older Spark versions. The trick is to look at the README.md git log and checkout the hopefully right commit that supports your Spark cluster version.

In our case for Spark 2.3 we used git commit bc82a57e4134452f19a11cd127bd4c6a25f75020.

On Gentoo, make sure to use dev-java/sbt-bin since the non binary version is vastly out of date and won’t build the project. You need at least version 1.3.

The scylla-migrator plan

The documentation explains that we need a config file that points to a source cluster+table and a destination cluster+table as long as they have the same schema structure…

Renaming is then as simple as duplicating the schema using CQLSH and running the migrator!

But what about our compacted version of our original table? The schema is different from the source table!…

Good news is that as long as all your columns remain present, you can also change the PRIMARY KEY of your destination table and it will still work!

This make the scylla-migrator an amazing tool to reshape or pivot tables!

  • the column date is renamed to seen_date: that’s okay, scylla-migrator supports column renaming (it’s a Spark dataframe after all)!
  • the PRIMARY KEY is different in the compacted table since we removed the ‘date‘ from the clustering columns: we’ll get a compacted table for free!

Using scylla-migrator

The documentation is a bit poor on how to submit your application to a Hadoop YARN cluster but that’s kind of expected.

It also did not mention how to connect to a SSL enabled cluster (are there people really not using SSL on the wire in their production environment?)… anyway let’s not start a flame war 🙂

The trick that will save you is to know that you can append all the usual Spark options that are available in the spark-cassandra-connector!

Submitting to a Kerberos protected Hadoop YARN cluster targeting a SSL enabled Scylla cluster then looks like this:

export JAR_NAME=target/scala-2.11/scylla-migrator-assembly-0.0.1.jar
export KRB_PRINCIPAL=USERNAME

spark2-submit \
 --name ScyllaMigratorApplication \
 --class com.scylladb.migrator.Migrator  \
 --conf spark.cassandra.connection.ssl.clientAuth.enabled=True  \
 --conf spark.cassandra.connection.ssl.enabled=True  \
 --conf spark.cassandra.connection.ssl.trustStore.path=jssecacerts  \
 --conf spark.cassandra.connection.ssl.trustStore.password=JKS_PASSWORD  \
 --conf spark.cassandra.input.consistency.level=LOCAL_QUORUM \
 --conf spark.cassandra.output.consistency.level=LOCAL_QUORUM \
 --conf spark.scylla.config=config.yaml \
 --conf spark.yarn.executor.memoryOverhead=1g \
 --conf spark.blacklist.enabled=true  \
 --conf spark.blacklist.task.maxTaskAttemptsPerExecutor=1  \
 --conf spark.blacklist.task.maxTaskAttemptsPerNode=1  \
 --conf spark.blacklist.stage.maxFailedTasksPerExecutor=1  \
 --conf spark.blacklist.stage.maxFailedExecutorsPerNode=1  \
 --conf spark.executor.cores=16 \
 --deploy-mode client \
 --files jssecacerts \
 --jars ${JAR_NAME}  \
 --keytab ${KRB_PRINCIPAL}.keytab  \
 --master yarn \
 --principal ${KRB_PRINCIPAL}  \
 ${JAR_NAME}

Note that we chose to apply a higher consistency level to our reads using a LOCAL_QUORUM instead of the default LOCAL_ONE. I strongly encourage you to do the same since it’s appropriate when you’re using this kind of tool!

Column renaming is simply expressed in the configuration file like this:

# Column renaming configuration.
renames:
  - from: date
    to: seen_date

Tuning scylla-migrator

While easy to use, tuning scylla-migrator to operate those migrations as fast as possible turned out to be a real challenge (remember we have some production applications shut down during the process).

Even using 300+ Spark executors I couldn’t get my Scylla cluster utilization to more than 50% and migrating a single table with a bit more than 1B rows took almost 2 hours…

We found the best knobs to play with thanks to the help of Lubos Kosco and this blog post from ScyllaDB:

  • Increase the splitCount setting: more splits means more Spark executors will be spawned and more tasks out of it. While it might be magic on a pure Spark deployment it’s not that amazing on a Hadoop YARN one where executors are scheduled in containers with 1 core by default. We simply moved it from 256 to 384.
  • Disable compaction on destination tables schemas. This gave us a big boost and saved the day since it avoids adding the overhead of compacting while you’re pushing down data hard!

To disable compaction on a table simply:

ALTER TABLE ids_by_partnerid_history WITH compaction = {'class': 'NullCompactionStrategy'};

Remember to run a manual compaction (nodetool compact <keyspace> <table>) and to enable compaction back on your tables once you’re done!

Happy Scylla tables mangling!

October 21 2020

DISTUTILS_USE_SETUPTOOLS, QA spam and… more QA spam?

Michał Górny (mgorny) October 21, 2020, 19:52

I suppose that most of the Gentoo developers have seen at least one of the ‘uses a probably incorrect DISTUTILS_USE_SETUPTOOLS value’ bugs by now. Over 350 have been filed so far, and new ones are filed practically daily. The truth is, I’ve never intended for this QA check to result in bugs being filed against packages, and certainly not that many bugs.

This is not an important problem to be fixed immediately. The vast majority of Python packages depend on setuptools at build time (this is why the build-time dependency is the eclass’ default), and being able to unmerge setuptools is not a likely scenario. The underlying idea was that the QA check would make it easier to update DISTUTILS_USE_SETUPTOOLS when bumping packages.

Nobody has asked me for my opinion, and now we have hundreds of bugs that are not very helpful. In fact, the effort involved in going through all the bugmail, updating packages and closing the bugs greatly exceeds the negligible gain. Nevertheless, some people actually did it. I have bad news for them: setuptools upstream has changed entry point mechanism, and most of the values will have to change again. Let me elaborate on that.

The current logic

The current eclass logic revolves around three primary values:

  • no indicating that the package does not use setuptools
  • bdepend indicating that the package uses setuptools at build time only
  • rdepend indicating that the package uses setuptools at build- and runtime

There’s also support for pyproject.toml but it’s tangential to the problem at hand, so let’s ignore it.

The setuptools package — besides the build system — includes a pkg_resources sub-package that can be used to access package’s metadata and resources. The two primary uses of rdepend revolves around this. These are:

  1. console_scripts entry points — i.e. autogenerated executable scripts that call a function within the installed package rather than containing the program code itself.
  2. Direct uses of pkg_resources in the modules installed by the package.

Both of these cases were equivalent from dependency standpoint. Well, not anymore.

Entry points via importlib.metadata

Well, the big deal is the importlib.metadata module that was added in Python 3.8 (there’s also a relevant importlib.resources module since Python 3.7). It is a built-in module that provides routines to access the installed package metadata, and therefore renders another part of pkg_resources redundant.

The big deal is that the new versions of setuptools have embraced it, and no longer require pkg_resources to run entry points. To be more precise, the new logic selects the built-in module as the first choice, with fallback to the importlib_metadata backport and finally to pkg_resources.

This means that the vast majority of packages that used to depend on setuptools at runtime, no longer does strictly that. With Python 3.8 and newer, they have no additional runtime dependencies and just require setuptools at build time. With older versions of Python, they prefer importlib_metadata over it. In both cases, the packages can still use pkg_resources directly though.

How to resolve it via the eclass?

Now, technically speaking this means replacing rdepend with three new variants:

  • scripts — that means build-time dependency on setuptools + runtime impl-conditional dep on importlib_metadata, for pure entry point usage.
  • rdepend — that means runtime dependency on setuptools, for pure pkg_resources usage.
  • scripts+rdepend — for packages that combine both.

Of course, this means that the existing packages would get a humongous number of new bug reports, often requesting a change to the value that was updated recently. The number could be smaller if we changed the existing meaning of rdepend to mean importlib.metadata, and introduced a new value for pkg_resources.

Still, that’s not the best part. The real fun idea is that once we remove Python 3.7, all Python versions would have importlib.metadata built-in and the distinction will no longer be necessary. Eventually, everyone would have to update the value again, this time to bdepend. Great, right?

…or not to resolve it?

Now that we’ve discussed the solution recommended to me, let’s consider an alternative. For the vast majority of packages, the runtime dependency on setuptools is unnecessary. If the user uses Python 3.8+ or has importlib_metadata installed (which is somewhat likely, due to direct dependencies on it), pkg_resources will not be used by the entry points. Nevertheless, setuptools is still pretty common as a build-time dependencies and, as I said before, it makes little sense to uninstall it.

We can simply keep things as-is. Sure, the dependencies will not be 100% optimal. Yet, the dependency on setuptools will ensure that entry points continue working even if the user does not have importlib_metadata installed. We will eventually want to update DISTUTILS_USE_SETUPTOOLS logic but we can wait for it till Python versions older than 3.8 become irrelevant, and we are back to three main variants.

I suppose that most of the Gentoo developers have seen at least one of the ‘uses a probably incorrect DISTUTILS_USE_SETUPTOOLS value’ bugs by now. Over 350 have been filed so far, and new ones are filed practically daily. The truth is, I’ve never intended for this QA check to result in bugs being filed against packages, and certainly not that many bugs.

This is not an important problem to be fixed immediately. The vast majority of Python packages depend on setuptools at build time (this is why the build-time dependency is the eclass’ default), and being able to unmerge setuptools is not a likely scenario. The underlying idea was that the QA check would make it easier to update DISTUTILS_USE_SETUPTOOLS when bumping packages.

Nobody has asked me for my opinion, and now we have hundreds of bugs that are not very helpful. In fact, the effort involved in going through all the bugmail, updating packages and closing the bugs greatly exceeds the negligible gain. Nevertheless, some people actually did it. I have bad news for them: setuptools upstream has changed entry point mechanism, and most of the values will have to change again. Let me elaborate on that.

The current logic

The current eclass logic revolves around three primary values:

  • no indicating that the package does not use setuptools
  • bdepend indicating that the package uses setuptools at build time only
  • rdepend indicating that the package uses setuptools at build- and runtime

There’s also support for pyproject.toml but it’s tangential to the problem at hand, so let’s ignore it.

The setuptools package — besides the build system — includes a pkg_resources sub-package that can be used to access package’s metadata and resources. The two primary uses of rdepend revolves around this. These are:

  1. console_scripts entry points — i.e. autogenerated executable scripts that call a function within the installed package rather than containing the program code itself.
  2. Direct uses of pkg_resources in the modules installed by the package.

Both of these cases were equivalent from dependency standpoint. Well, not anymore.

Entry points via importlib.metadata

Well, the big deal is the importlib.metadata module that was added in Python 3.8 (there’s also a relevant importlib.resources module since Python 3.7). It is a built-in module that provides routines to access the installed package metadata, and therefore renders another part of pkg_resources redundant.

The big deal is that the new versions of setuptools have embraced it, and no longer require pkg_resources to run entry points. To be more precise, the new logic selects the built-in module as the first choice, with fallback to the importlib_metadata backport and finally to pkg_resources.

This means that the vast majority of packages that used to depend on setuptools at runtime, no longer does strictly that. With Python 3.8 and newer, they have no additional runtime dependencies and just require setuptools at build time. With older versions of Python, they prefer importlib_metadata over it. In both cases, the packages can still use pkg_resources directly though.

How to resolve it via the eclass?

Now, technically speaking this means replacing rdepend with three new variants:

  • scripts — that means build-time dependency on setuptools + runtime impl-conditional dep on importlib_metadata, for pure entry point usage.
  • rdepend — that means runtime dependency on setuptools, for pure pkg_resources usage.
  • scripts+rdepend — for packages that combine both.

Of course, this means that the existing packages would get a humongous number of new bug reports, often requesting a change to the value that was updated recently. The number could be smaller if we changed the existing meaning of rdepend to mean importlib.metadata, and introduced a new value for pkg_resources.

Still, that’s not the best part. The real fun idea is that once we remove Python 3.7, all Python versions would have importlib.metadata built-in and the distinction will no longer be necessary. Eventually, everyone would have to update the value again, this time to bdepend. Great, right?

…or not to resolve it?

Now that we’ve discussed the solution recommended to me, let’s consider an alternative. For the vast majority of packages, the runtime dependency on setuptools is unnecessary. If the user uses Python 3.8+ or has importlib_metadata installed (which is somewhat likely, due to direct dependencies on it), pkg_resources will not be used by the entry points. Nevertheless, setuptools is still pretty common as a build-time dependencies and, as I said before, it makes little sense to uninstall it.

We can simply keep things as-is. Sure, the dependencies will not be 100% optimal. Yet, the dependency on setuptools will ensure that entry points continue working even if the user does not have importlib_metadata installed. We will eventually want to update DISTUTILS_USE_SETUPTOOLS logic but we can wait for it till Python versions older than 3.8 become irrelevant, and we are back to three main variants.

October 06 2020

Speeding up emerge depgraph calculation using PyPy3

Michał Górny (mgorny) October 06, 2020, 8:01

WARNING: Some of the respondents were< not able to reproduce my results. It is possible that this dependent on the hardware or even a specific emerge state. Please do not rely on my claims that PyPy3 runs faster, and verify it on your system before switching permanently.

If you used Gentoo for some time, you’ve probably noticed that emerge is getting slower and slower. Before I switched to SSD, my emerge could take even 10 minutes before it figured out what to do! Even now it’s pretty normal for the dependency calculation to take 2 minutes. Georgy Yakovlev recently tested PyPy3 on PPC64, and noticed a great speedup, apparently due to very poor optimization of CPython on that platform. I’ve attempted the same on amd64, and measured a 35% speedup nevertheless.

PyPy is an alternative implementation of Python that uses a JIT compiler to run Python code. JIT can achieve greater performance on computation-intensive tasks, at the cost of slower program startup. This means that it could be slower for some programs, and faster for others. In case of emerge dependency calculation, it’s definitely faster. A quick benchmark done using dev-perl/Dumbbench (great tool, by the way) shows, for today’s @world upgrade:

  • Python 3.9.0: 111.42 s ± 0.87 s (0.8%)
  • PyPy3.7 7.3.2: 72.30 s ± 0.23 s (0.3%)

dev-python/pypy3 is supported on Gentoo, on amd64, arm64, ppc64 and x86 targets. The interpreter itself takes quite a while to build (35­­–45 minutes on a modern Ryzen), so you may want to suggest emerge to grab dev-python/pypy3-exe-bin:

$ emerge -nv dev-python/pypy3 dev-python/pypy3-exe-bin

If you want to build it from source, it is recommended to grab dev-python/pypy first (possibly with dev-python/pypy-exe-bin for faster bootstrap), as building with PyPy itself is much faster:

# use prebuilt compiler for fast bootstrap
$ emerge -1v dev-python/pypy dev-python/pypy-exe-bin
# rebuild the interpreter
$ emerge -nv dev-python/pypy dev-python/pypy-exe
# build pypy3
$ emerge -nv dev-python/pypy3

Update 2020-10-07: Afterwards, you need to rebuild Portage and its dependencies with PyPy3 support enabled. The easiest way of doing it is to enable the PyPy3 target globally, and rebuilding relevant packages:

$ echo '*/* PYTHON_TARGETS: pypy3' >> /etc/portage/package.use
$ emerge -1vUD sys-apps/portage

Finally, you can use python-exec’s per-program configuration to use PyPy3 for emerge while continuing to use CPython for other programs:

$ echo pypy3 >> /etc/python-exec/emerge.conf
# yep, that's pypy3.7
$ emerge --info | head -1
Portage 3.0.7 (python 3.7.4-final-0, default/linux/amd64/17.1/desktop, gcc-9.3.0, glibc-2.32-r2, 5.8.12 x86_64)

WARNING: Some of the respondents were< not able to reproduce my results. It is possible that this dependent on the hardware or even a specific emerge state. Please do not rely on my claims that PyPy3 runs faster, and verify it on your system before switching permanently.

If you used Gentoo for some time, you’ve probably noticed that emerge is getting slower and slower. Before I switched to SSD, my emerge could take even 10 minutes before it figured out what to do! Even now it’s pretty normal for the dependency calculation to take 2 minutes. Georgy Yakovlev recently tested PyPy3 on PPC64, and noticed a great speedup, apparently due to very poor optimization of CPython on that platform. I’ve attempted the same on amd64, and measured a 35% speedup nevertheless.

PyPy is an alternative implementation of Python that uses a JIT compiler to run Python code. JIT can achieve greater performance on computation-intensive tasks, at the cost of slower program startup. This means that it could be slower for some programs, and faster for others. In case of emerge dependency calculation, it’s definitely faster. A quick benchmark done using dev-perl/Dumbbench (great tool, by the way) shows, for today’s @world upgrade:

  • Python 3.9.0: 111.42 s ± 0.87 s (0.8%)
  • PyPy3.7 7.3.2: 72.30 s ± 0.23 s (0.3%)

dev-python/pypy3 is supported on Gentoo, on amd64, arm64, ppc64 and x86 targets. The interpreter itself takes quite a while to build (35­­–45 minutes on a modern Ryzen), so you may want to suggest emerge to grab dev-python/pypy3-exe-bin:

$ emerge -nv dev-python/pypy3 dev-python/pypy3-exe-bin

If you want to build it from source, it is recommended to grab dev-python/pypy first (possibly with dev-python/pypy-exe-bin for faster bootstrap), as building with PyPy itself is much faster:

# use prebuilt compiler for fast bootstrap
$ emerge -1v dev-python/pypy dev-python/pypy-exe-bin
# rebuild the interpreter
$ emerge -nv dev-python/pypy dev-python/pypy-exe
# build pypy3
$ emerge -nv dev-python/pypy3

Update 2020-10-07: Afterwards, you need to rebuild Portage and its dependencies with PyPy3 support enabled. The easiest way of doing it is to enable the PyPy3 target globally, and rebuilding relevant packages:

$ echo '*/* PYTHON_TARGETS: pypy3' >> /etc/portage/package.use
$ emerge -1vUD sys-apps/portage

Finally, you can use python-exec’s per-program configuration to use PyPy3 for emerge while continuing to use CPython for other programs:

$ echo pypy3 >> /etc/python-exec/emerge.conf
# yep, that's pypy3.7
$ emerge --info | head -1
Portage 3.0.7 (python 3.7.4-final-0, default/linux/amd64/17.1/desktop, gcc-9.3.0, glibc-2.32-r2, 5.8.12 x86_64)

September 16 2020

Console-bound systemd services, the right way

Marek Szuba (marecki) September 16, 2020, 17:40

Let’s say that you need to run on your system some sort server software which instead of daemonising, has a command console permanently attached to standard input. Let us also say that said console is the only way for the administrator to interact with the service, including requesting its orderly shutdown – whoever has written it has not implemented any sort of signal handling so sending SIGTERM to the service process causes it to simply drop dead, potentially losing data in the process. And finally, let us say that the server in question is proprietary software so it isn’t really possible for you to fix any of the above in the source code (yes, I am talking about a specific piece of software – which by the way is very much alive and kicking as of late 2020). What do you do?

According to the collective wisdom of World Wide Web, the answer to this question is “use a terminal multiplexer like tmux or screen“, or at the very least a stripped-down variant of same such as dtach. OK, that sort of works – what if you want to run it as a proper system-managed service under e.g. OpenRC? The answer of the Stack Exchange crowd: have your init script invoke the terminal multiplexer. Oooooookay, how about under systemd, which actually prefers services it manages not to daemonise by itself? Nope, still “use a terminal multiplexer”.

What follows is my attempt to run a service like this under systemd more efficiently and elegantly, or at least with no extra dependencies beyond basic Unix shell commands.

Let us have a closer look at what systemd does with standard I/O of processes it spawns. The man page systemd.exec(5) tells us that what happens here is controlled by the directives StandardInput, StandardOutput and StandardError. By default the former is assigned to null while the latter two get piped to the journal, there are however quite a few other options here. According to the documentation, here is what systemd allows us to connect to standard input:

    • we are not interested in null (for obvious reasons) or any of the tty options (the whole point of this exercise is to run fully detached from any terminals);
    • data would work if we needed to feed some commands to the service when it starts but is useless for triggering a shutdown;
    • file looks promising – just point it to a FIFO on the file system and we’re all set – but it doesn’t actually take care of creating the FIFO for us. While we could in theory work around that by invoking mkfifo (and possibly chown if the service is to run as a specific user) in ExecStartPre, let’s see if we can find a better option
    • socket “is valid in socket-activated services only” and the corresponding socket unit must “have Accept=yes set”. What we want is the opposite, i.e. for the service to create its socket
    • finally, there is fd – which seems to be exactly what we need. According to the documentation all we have to do is write a socket unit creating a FIFO with appropriate ownership and permissions, make it a dependency of our service using the Sockets directive, and assign the corresponding named file descriptor to standard input.

Let’s try it out. To begin with, our socket unit “proprietarycrapd.socket”. Note that I have successfully managed to get this to work using unit templates as well, %i expansion works fine both here and while specifying unit or file-descriptor names in the service unit – but in order to avoid any possible confusion caused by the fact socket-activated services explicitly require being defined with templates, I have based my example on static units:

[Unit]
Description=Command FIFO for proprietarycrapd

[Socket]
ListenFIFO=/run/proprietarycrapd/pcd.control
DirectoryMode=0700
SocketMode=0600
SocketUser=pcd
SocketGroup=pcd
RemoveOnStop=true

Apart from the fact the unit in question has got no [Install] section (which makes sense given we want this socket to only be activated by the corresponding service, not by systemd itself), nothing out of the ordinary here. Note that since we haven’t used the directive FileDescriptorName, systemd will apply default behaviour and give the file descriptor associated with the FIFO the name of the socket unit itself.

And now, our service unit “proprietarycrapd.service”:

[Unit]
Description=proprietarycrap daemon
After=network.target

[Service]
User=pcd
Group=pcd
Sockets=proprietarycrapd.socket
StandardInput=socket
StandardOutput=journal
StandardError=journal
ExecStart=/opt/proprietarycrap/bin/proprietarycrapd
ExecStop=/usr/local/sbin/proprietarycrapd-stop

[Install]
WantedBy=multi-user.target

StandardInput=socket??? Whatever’s happened to StandardInput=fd:proprietarycrapd.socket??? Here is an odd thing. If I use the latter on my system, the service starts fine and gets the FIFO attached to its standard input – but when I try to stop the service the journal shows “Failed to load a named file descriptor: No such file or directory”, the ExecStop command is not run and systemd immediately fires a SIGTERM at the process. No idea why. Anyway, through trial and error I have found out that StandardInput=socket not only works fine in spite of being used in a service that is not socket-activated but actually does exactly what I wanted to achieve – so that is what I have ended up using.

Which brings us to the final topic, the ExecStop command. There are three reasons why I have opted for putting all the commands required to shut the server down in a shell script:

    • first and foremost, writing the shutdown command to the FIFO will return right away even if the service takes time to shut down. systemd sends SIGTERM to the unit process as soon as the last ExecStop command has exited so we have to follow the echo with something that waits for the server process to finish (see below)
    • systemd does not execute Exec commands in a shell so simply running echo > /run/proprietarycrapd/pcd.control doesn’t work, we would have to wrap the echo call in an explicit invocation of a shell
    • between the aforementioned two reasons and the fact the particular service for which I have created these units actually requires several commands in order to execute an orderly shutdown, I have decided that putting all those command in a script file instead of cramming them into the unit would be much cleaner.

The shutdown script itself is mostly unremarkable so I’ll only quote the bit responsible for waiting for the server to actually shut down. At present I am still looking for doing it in blocking fashion without adding more dependencies (wait only works on child processes of the current shell, the server in question does not create any lock files to which I could attach inotifywait, and attaching the latter to the relevant directory in /proc does not work) but in the meantime, the loop

while kill -0 “${MAINPID}” 2> /dev/null; do
sleep 1s
done

keeps the script ticking along until either the process has exited or the script has timed out (see the TimeoutStopSec directive in systemd.service(5)) and systemd has killed both it and the service itself.

Acknowledgements: with many thanks to steelman for having figured out the StandardInput=socket bit in particular and having let me bounce my ideas off him in general.

Let’s say that you need to run on your system some sort server software which instead of daemonising, has a command console permanently attached to standard input. Let us also say that said console is the only way for the administrator to interact with the service, including requesting its orderly shutdown – whoever has written it has not implemented any sort of signal handling so sending SIGTERM to the service process causes it to simply drop dead, potentially losing data in the process. And finally, let us say that the server in question is proprietary software so it isn’t really possible for you to fix any of the above in the source code (yes, I am talking about a specific piece of software – which by the way is very much alive and kicking as of late 2020). What do you do?

According to the collective wisdom of World Wide Web, the answer to this question is “use a terminal multiplexer like tmux or screen“, or at the very least a stripped-down variant of same such as dtach. OK, that sort of works – what if you want to run it as a proper system-managed service under e.g. OpenRC? The answer of the Stack Exchange crowd: have your init script invoke the terminal multiplexer. Oooooookay, how about under systemd, which actually prefers services it manages not to daemonise by itself? Nope, still “use a terminal multiplexer”.

What follows is my attempt to run a service like this under systemd more efficiently and elegantly, or at least with no extra dependencies beyond basic Unix shell commands.

Let us have a closer look at what systemd does with standard I/O of processes it spawns. The man page systemd.exec(5) tells us that what happens here is controlled by the directives StandardInput, StandardOutput and StandardError. By default the former is assigned to null while the latter two get piped to the journal, there are however quite a few other options here. According to the documentation, here is what systemd allows us to connect to standard input:

    • we are not interested in null (for obvious reasons) or any of the tty options (the whole point of this exercise is to run fully detached from any terminals);
    • data would work if we needed to feed some commands to the service when it starts but is useless for triggering a shutdown;
    • file looks promising – just point it to a FIFO on the file system and we’re all set – but it doesn’t actually take care of creating the FIFO for us. While we could in theory work around that by invoking mkfifo (and possibly chown if the service is to run as a specific user) in ExecStartPre, let’s see if we can find a better option
    • socket “is valid in socket-activated services only” and the corresponding socket unit must “have Accept=yes set”. What we want is the opposite, i.e. for the service to create its socket
    • finally, there is fd – which seems to be exactly what we need. According to the documentation all we have to do is write a socket unit creating a FIFO with appropriate ownership and permissions, make it a dependency of our service using the Sockets directive, and assign the corresponding named file descriptor to standard input.

Let’s try it out. To begin with, our socket unit “proprietarycrapd.socket”. Note that I have successfully managed to get this to work using unit templates as well, %i expansion works fine both here and while specifying unit or file-descriptor names in the service unit – but in order to avoid any possible confusion caused by the fact socket-activated services explicitly require being defined with templates, I have based my example on static units:

[Unit]
Description=Command FIFO for proprietarycrapd

[Socket]
ListenFIFO=/run/proprietarycrapd/pcd.control
DirectoryMode=0700
SocketMode=0600
SocketUser=pcd
SocketGroup=pcd
RemoveOnStop=true

Apart from the fact the unit in question has got no [Install] section (which makes sense given we want this socket to only be activated by the corresponding service, not by systemd itself), nothing out of the ordinary here. Note that since we haven’t used the directive FileDescriptorName, systemd will apply default behaviour and give the file descriptor associated with the FIFO the name of the socket unit itself.

And now, our service unit “proprietarycrapd.service”:

[Unit]
Description=proprietarycrap daemon
After=network.target

[Service]
User=pcd
Group=pcd
Sockets=proprietarycrapd.socket
StandardInput=socket
StandardOutput=journal
StandardError=journal
ExecStart=/opt/proprietarycrap/bin/proprietarycrapd
ExecStop=/usr/local/sbin/proprietarycrapd-stop

[Install]
WantedBy=multi-user.target

StandardInput=socket??? Whatever’s happened to StandardInput=fd:proprietarycrapd.socket??? Here is an odd thing. If I use the latter on my system, the service starts fine and gets the FIFO attached to its standard input – but when I try to stop the service the journal shows “Failed to load a named file descriptor: No such file or directory”, the ExecStop command is not run and systemd immediately fires a SIGTERM at the process. No idea why. Anyway, through trial and error I have found out that StandardInput=socket not only works fine in spite of being used in a service that is not socket-activated but actually does exactly what I wanted to achieve – so that is what I have ended up using.

Which brings us to the final topic, the ExecStop command. There are three reasons why I have opted for putting all the commands required to shut the server down in a shell script:

    • first and foremost, writing the shutdown command to the FIFO will return right away even if the service takes time to shut down. systemd sends SIGTERM to the unit process as soon as the last ExecStop command has exited so we have to follow the echo with something that waits for the server process to finish (see below)
    • systemd does not execute Exec commands in a shell so simply running echo > /run/proprietarycrapd/pcd.control doesn’t work, we would have to wrap the echo call in an explicit invocation of a shell
    • between the aforementioned two reasons and the fact the particular service for which I have created these units actually requires several commands in order to execute an orderly shutdown, I have decided that putting all those command in a script file instead of cramming them into the unit would be much cleaner.

The shutdown script itself is mostly unremarkable so I’ll only quote the bit responsible for waiting for the server to actually shut down. At present I am still looking for doing it in blocking fashion without adding more dependencies (wait only works on child processes of the current shell, the server in question does not create any lock files to which I could attach inotifywait, and attaching the latter to the relevant directory in /proc does not work) but in the meantime, the loop

while kill -0 “${MAINPID}” 2> /dev/null; do
sleep 1s
done

keeps the script ticking along until either the process has exited or the script has timed out (see the TimeoutStopSec directive in systemd.service(5)) and systemd has killed both it and the service itself.

Acknowledgements: with many thanks to steelman for having figured out the StandardInput=socket bit in particular and having let me bounce my ideas off him in general.

September 15 2020

Distribution kernel for Gentoo

Gentoo News (GentooNews) September 15, 2020, 5:00

The Gentoo Distribution Kernel project is excited to announce that our new Linux Kernel packages are ready for a wide audience! The project aims to create a better Linux Kernel maintenance experience by providing ebuilds that can be used to configure, compile, and install a kernel entirely through the package manager as well as prebuilt binary kernels. We are currently shipping three kernel packages:

  • sys-kernel/gentoo-kernel - providing a kernel with genpatches applied, built using the package manager with either a distribution default or a custom configuration
  • sys-kernel/gentoo-kernel-bin - prebuilt version of gentoo-kernel, saving time on compiling
  • sys-kernel/vanilla-kernel - providing a vanilla (unmodified) upstream kernel

All the packages install the kernel as part of the package installation process — just like the rest of your system! More information can be found in the Gentoo Handbook and on the Distribution Kernel project page. Happy hacking!

Larry with Tux as cowboy

The Gentoo Distribution Kernel project is excited to announce that our new Linux Kernel packages are ready for a wide audience! The project aims to create a better Linux Kernel maintenance experience by providing ebuilds that can be used to configure, compile, and install a kernel entirely through the package manager as well as prebuilt binary kernels. We are currently shipping three kernel packages:

All the packages install the kernel as part of the package installation process — just like the rest of your system! More information can be found in the Gentoo Handbook and on the Distribution Kernel project page. Happy hacking!

VIEW

SCOPE

FILTER
  from
  to