August 02 2020

Why proactively clean Python 2 up?

Michał Górny (mgorny) August 02, 2020, 10:18

It seems a recurring complaint that we’re too aggressive on cleaning Python 2 up from packages. Why remove it if (package’s) upstream still supports py2? Why remove it when it still works? Why remove it when somebody’s ready to put some work to keep it working?

I’m pretty sure that you’re aware that Python 2 has finally reached its end-of-life. It’s past its last release, and the current version is most likely vulnerable. We know we can’t remove it entirely just yet (but the clock is ticking!), so why remove its support here and there instead of keeping it some more?

This is best explained on the example of dev-python/twisted — but dev-python/pillow is also quite similar. Twisted upstream removed support for Python 2 at version 20. This means that we ended up having to keep two versions of Twisted — 19 that still supports Python 2, and 20 that does not. What does that means for our users?

Firstly, they can’t normally upgrade Twisted if at least one of its reverse dependencies supports Python 2 and is installed. What’s important is that the user does not have to meaningfully need or use Python 2 in that reverse dependency. It is entirely sufficient that it supports Python 2 and the user is using default PYTHON_TARGETS.

Of course, you could argue that changing the default PYTHON_TARGETS would resolve the problem without having to proactively remove Python 2 from Twisted revdeps. Today, I’m not sure which of the two options is better. However, back when cleanup started changing default PT would involve a lot of pain for the users. We’d have to reenable 2.7 via package.use for many packages (but which ones?) or the users would have to reenable it themselves. But that’s really tangential now.

Secondly, when upstream stops supporting the old version, the maintenance cost rises quickly. Since we don’t allow mixing two versions easily (and I don’t really want to go down that path), a single version must provide all implementations that the union of its reverse dependencies requires. This meant that I had to put significant effort fixing Python 3.8 and 3.9 support in Twisted 19.

Thirdly, old versions tend to end up becoming vulnerable. This is now the case both with Twisted and Pillow! In both cases, we can’t clean up vulnerable versions yet because they still have unresolved Python 2 reverse dependencies. We have a pretty descriptive phrase for this kind of situation in Polish: «to arouse with your hand in the potty».

What’s my point here? Removing Python 2 proactively means removing it at our leisure. We start with packages that don’t need it (because they fully support Python 3), we unlock the removal in their dependencies, we clean these dependencies… and when one of the upstreams decides to remove it, we don’t have to do anything because we’ve already done that and resolved all the issues. And we don’t have to worry about having to quickly clean up the depgraph and remove vulnerable versions or perform non-trivial backports.

It seems a recurring complaint that we’re too aggressive on cleaning Python 2 up from packages. Why remove it if (package’s) upstream still supports py2? Why remove it when it still works? Why remove it when somebody’s ready to put some work to keep it working?

I’m pretty sure that you’re aware that Python 2 has finally reached its end-of-life. It’s past its last release, and the current version is most likely vulnerable. We know we can’t remove it entirely just yet (but the clock is ticking!), so why remove its support here and there instead of keeping it some more?

This is best explained on the example of dev-python/twisted — but dev-python/pillow is also quite similar. Twisted upstream removed support for Python 2 at version 20. This means that we ended up having to keep two versions of Twisted — 19 that still supports Python 2, and 20 that does not. What does that means for our users?

Firstly, they can’t normally upgrade Twisted if at least one of its reverse dependencies supports Python 2 and is installed. What’s important is that the user does not have to meaningfully need or use Python 2 in that reverse dependency. It is entirely sufficient that it supports Python 2 and the user is using default PYTHON_TARGETS.

Of course, you could argue that changing the default PYTHON_TARGETS would resolve the problem without having to proactively remove Python 2 from Twisted revdeps. Today, I’m not sure which of the two options is better. However, back when cleanup started changing default PT would involve a lot of pain for the users. We’d have to reenable 2.7 via package.use for many packages (but which ones?) or the users would have to reenable it themselves. But that’s really tangential now.

Secondly, when upstream stops supporting the old version, the maintenance cost rises quickly. Since we don’t allow mixing two versions easily (and I don’t really want to go down that path), a single version must provide all implementations that the union of its reverse dependencies requires. This meant that I had to put significant effort fixing Python 3.8 and 3.9 support in Twisted 19.

Thirdly, old versions tend to end up becoming vulnerable. This is now the case both with Twisted and Pillow! In both cases, we can’t clean up vulnerable versions yet because they still have unresolved Python 2 reverse dependencies. We have a pretty descriptive phrase for this kind of situation in Polish: «to arouse with your hand in the potty».

What’s my point here? Removing Python 2 proactively means removing it at our leisure. We start with packages that don’t need it (because they fully support Python 3), we unlock the removal in their dependencies, we clean these dependencies… and when one of the upstreams decides to remove it, we don’t have to do anything because we’ve already done that and resolved all the issues. And we don’t have to worry about having to quickly clean up the depgraph and remove vulnerable versions or perform non-trivial backports.

July 20 2020

Updated Gentoo RISC-V stages

Andreas K. Hüttel (dilfridge) July 20, 2020, 16:28
♦I finally got around to updating the experimental riscv stages. You can find the result on our webserver. All stages use the rv64gc instruction set; there is a multilib stage with both lp64 and lp64d support, and there are non-multilib stages for both lp64 and lp64d ABI. Please test, and report bugs if anything doesn't work.
As for the technical details, the stages are built using qemu-user on a big and beefy Gentoo amd64 AWS instance. We are currently working on automating that process, such that riscv (and potentially also arm and others) get the same level of support as amd64 and friends. Thanks a lot to Amazon for the credits via their open source promotial program!
I finally got around to updating the experimental riscv stages. You can find the result on our webserver. All stages use the rv64gc instruction set; there is a multilib stage with both lp64 and lp64d support, and there are non-multilib stages for both lp64 and lp64d ABI. Please test, and report bugs if anything doesn't work.
As for the technical details, the stages are built using qemu-user on a big and beefy Gentoo amd64 AWS instance. We are currently working on automating that process, such that riscv (and potentially also arm and others) get the same level of support as amd64 and friends. Thanks a lot to Amazon for the credits via their open source promotial program!

July 07 2020

Gentoo on Android 64bit Release

Gentoo News (GentooNews) July 07, 2020, 5:00

Gentoo Project Android is pleased to announce a new 64bit release of the stage3 Android prefix tarball. This is a major release after 2.5 years of development, featuring gcc-10.1.0, binutils-2.34 and glibc-2.31. Enjoy Gentoo in your pocket!

gentoo-android logo

Gentoo Project Android is pleased to announce a new 64bit release of the stage3 Android prefix tarball. This is a major release after 2.5 years of development, featuring gcc-10.1.0, binutils-2.34 and glibc-2.31. Enjoy Gentoo in your pocket!

July 04 2020

gentoo tinderbox

Agostino Sarubbo (ago) July 04, 2020, 13:03

If you are visiting this page, it is very likely that the software you maintain has been analyzed by my tinderbox system.

What is a tinderbox?

It is a machine that compiles 24/7 that aims to find build failures, test failures, QA issues and so on in the portage tree.
It can be differentiated into:

– tinderbox – ci


TINDERBOX:

It compiles the entire portage tree against a particular change like:
– a new GCC version
– a new LDFLAG
– a different toolchain
– and so on

In short it uses uncommon but supported settings and looks for breakage.

CI:

It is a continuous integration; the CI system compiles the packages after they have been touched in gentoo.git

The CI system uses a standard set of settings, so if you get a bug report from it, it is very likely that the failure is reproducible for users too.

What are the rules that you may know when you see a report from those systems?

1) The reports are filed automatically.
2) Because of the first, it is not possible for me to set an exact error in the bug summary. Instead a general error is used.
3) Because of the above, maintainer is encouraged to set an appropriate summary at its convenience
4) Because of the first, I don’t know if additional files are needed (e.g. test-suite.log testlog.txt CMakeOutput.log CMakeError.log LastTest.log config.log testsuite.log autoconf.out). So if you need anything else please ask for them.
5) If you ask for another log, I have to stop the tinderbox service, so there may be a delay between your request and my reaction.
6) There may be an internal reference between round brackets on the “Discovered on” line. This is for me to understand where that failure was reproduced.
7) If you see ‘ci’ as internal reference after you pushed a fix, it is very probably that the bug still exist, or there is another failure in the same ebuild phase. Please inspect deeply the build log. Point 8 may help you about that.
8) At the beginning of the build log a git SHA of the repository at the time of emerging is provided. For convenience there is a link.
9) To avoid making a separate attachment on bugzilla, at the end of the build log there is the ’emerge –info’, please check it DEEPLY to understand the system configuration and what differs respect to a more ‘standard’ system.
10) If you see a compressed build log, is because the plain text version exceeds the limits on our bugzilla (1MB).
11) This system is not perfect. There may be duplicates or invalid bugs.
12) My best suggestion is try to reproduce the issue on empty stage3 (or docker for convenience).
13) When you close the bug with a resolution different from RESOLVED/FIXED please not be cryptic.
14) If new points will be added, there may be a mention like “Valid from YY:MM:DD”

How to fix the common errors:

1) Compile/build failure:
It depends on the error. Please get in touch with upstream if you are unsure.

2) Test failure:
It depends on the error. Please get in touch with upstream if you are unsure.

3) CFLAGS/LDFLAGS not respected:
You can touch the build system or inject the flags in the ebuild where possible. There are a lot of examples in the tracker.

4) -Wformat-security failure:
TBD

5) Metainfo installed in /usr/share/appdata:
Install metainfo into /usr/share instead of /usr/share/appdata

6) Python modules that are not byte-compiled
TBD

7) Unrecognized configure options:
Remove the configure options from the ebuild where possible. Sometimes there are false positives related to the option passed to configure in subdirectories.

8) Compressed manpages and documentation:
Decompress documentation and install it as plain text.

9) Icon cache not updated:
TBD

10) Deprecated configure.in:
TBD

11) .Desktop do not pass validation:
TBD

12) Path that should be created at runtime:
TBD

13) Libraries that lack NEEDED entries:
TBD

14) Libraries that lack a SONAME:
TBD

15) Text relocation:
TBD

16) Toolchain binaries called directly (cc/gcc/g++/c++/nm/ar/ranlib/cpp/ld/strip/objcopy/objdump/size/as/strings/readelf and so on):
TBD

17) Files with name not encoded with UTF-8:
TBD

18) Files with broken symlink:
TBD

19) Command that do not exist:
TBD

20) Pkg-config files with wrong LDFLAGS:
TBD

21) Pre-stripped files:
TBD

22) File collision:
TBD

23) Compile failure if CPP is set to CC -E:
TBD

24) Compile failure with -fno-common:
TBD

25) Files with unresolved SONAME dependencies:
TBD

26) Files that contain insecure RUNPATHs:
TBD

27) Files installed into unexpected paths:
TBD

28) LD usage instead of CC/CXX:
TBD

29) Link failure with LLD because of /usr/lib:
TBD

30) Compilation in src_install phase:
TBD

31) Automake usage in maintainer-mode:
TBD

32) Mimeinfo cache not updated when .desktop files with MimeType are installed:
TBD

33) Broken png files installed:
TBD

34) Mime-info files installed without update mime-info cache:
TBD

35) Udev rules installed into wrong directory:
TBD

ago (ago ) July 04, 2020, 13:03

If you are visiting this page, it is very likely that the software you maintain has been analyzed by my tinderbox system.

What is a tinderbox?

It is a machine that compiles 24/7 that aims to find build failures, test failures, QA issues and so on in the portage tree.
It can be differentiated into:

– tinderbox

– ci

TINDERBOX:

It compiles the entire portage tree against a particular change like:
– a new GCC version
– a new LDFLAG
– a different toolchain
– and so on

In short it uses uncommon but supported settings and looks for breakage.

CI:

It is a continuous integration; the CI system compiles the packages after they have been touched in gentoo.git

The CI system uses a standard set of settings, so if you get a bug report from it, it is very likely that the failure is reproducible for users too.

What are the rules that you may know when you see a report from those systems?

1) The reports are filed automatically.
2) Because of the first, it is not possible for me to set an exact error in the bug summary. Instead a general error is used.
3) Because of the above, maintainer is encouraged to set an appropriate summary at its convenience
4) Because of the first, I don’t know if additional files are needed (e.g. test-suite.log testlog.txt CMakeOutput.log CMakeError.log LastTest.log config.log testsuite.log autoconf.out). So if you need anything else please ask for them.
5) If you ask for another log, I have to stop the tinderbox service, so there may be a delay between your request and my reaction.
6) There may be an internal reference between round brackets on the “Discovered on” line. This is for me to understand where that failure was reproduced.
7) If you see ‘ci’ as internal reference after you pushed a fix, it is very probably that the bug still exist, or there is another failure in the same ebuild phase. Please inspect deeply the build log. Point 8 may help you about that.
8) At the beginning of the build log a git SHA of the repository at the time of emerging is provided. For convenience there is a link.
9) To avoid making a separate attachment on bugzilla, at the end of the build log there is the ’emerge –info’, please check it DEEPLY to understand the system configuration and what differs respect to a more ‘standard’ system.
10) If you see a compressed build log, is because the plain text version exceeds the limits on our bugzilla (1MB).
11) This system is not perfect. There may be duplicates or invalid bugs.
12) My best suggestion is try to reproduce the issue on empty stage3 (or docker for convenience).
13) When you close the bug with a resolution different from RESOLVED/FIXED please not be cryptic.
14) If new points will be added, there may be a mention like “Valid from YY:MM:DD”

How to fix the common errors:

1) Compile/build failure:
It depends on the error. Please get in touch with upstream if you are unsure.

2) Test failure:
It depends on the error. Please get in touch with upstream if you are unsure.

3) CFLAGS/LDFLAGS not respected:
You can touch the build system or inject the flags in the ebuild where possible. There are a lot of examples in the tracker.

4) -Wformat-security failure:
TBD

5) Metainfo installed in /usr/share/appdata:
Install metainfo into /usr/share instead of /usr/share/appdata

6) Python modules that are not byte-compiled
TBD

7) Unrecognized configure options:
Remove the configure options from the ebuild where possible. Sometimes there are false positives related to the option passed to configure in subdirectories.

8) Compressed manpages and documentation:
Decompress documentation and install it as plain text.

9) Icon cache not updated:
TBD

10) Deprecated configure.in:
TBD

11) .Desktop do not pass validation:
TBD

12) Path that should be created at runtime:
TBD

13) Libraries that lack NEEDED entries:
TBD

14) Libraries that lack a SONAME:
TBD

15) Text relocation:
TBD

16) Toolchain binaries called directly (cc/gcc/g++/c++/nm/ar/ranlib/cpp/ld/strip/objcopy/objdump/size/as/strings/readelf and so on):
TBD

17) Files with name not encoded with UTF-8:
TBD

18) Files with broken symlink:
TBD

19) Command that do not exist:
TBD

20) Pkg-config files with wrong LDFLAGS:
TBD

21) Pre-stripped files:
TBD

22) File collision:
TBD

23) Compile failure if CPP is set to CC -E:
TBD

24) Compile failure with -fno-common:
TBD

25) Files with unresolved SONAME dependencies:
TBD

26) Files that contain insecure RUNPATHs:
TBD

27) Files installed into unexpected paths:
TBD

28) LD usage instead of CC/CXX:
TBD

29) Link failure with LLD because of /usr/lib:
TBD

30) Compilation in src_install phase:
TBD

31) Automake usage in maintainer-mode:
TBD

32) Mimeinfo cache not updated when .desktop files with MimeType are installed:
TBD

33) Broken png files installed:
TBD

34) Mime-info files installed without update mime-info cache:
TBD

35) Udev rules installed into wrong directory:
TBD

Official Gentoo Docker images

Gentoo News (GentooNews) July 04, 2020, 5:00

Did you already know that we have official Gentoo Docker images available on Docker Hub?! The most popular one is based on the amd64 stage. Images are created automatically; you can peek at the source code for this on our git server. Thanks to the Gentoo Docker project!

docker logo

Did you already know that we have official Gentoo Docker images available on Docker Hub?! The most popular one is based on the amd64 stage. Images are created automatically; you can peek at the source code for this on our git server. Thanks to the Gentoo Docker project!

June 04 2020

Baïkal (CalDAV) 0.7.0 in Gentoo

Nathan Zachary (nathanzachary) June 04, 2020, 3:30

Just this past week, the new version of of Baïkal (0.7.0)—a PHP CalDAV and CardDAV server based on Sabre—was released, and one of the key changes was that support was added for more modern versions of PHP (like 7.4).

Since my personal Gentoo server is running the ~amd64 branch, I had to wait for this release in order to get my CalDAV server up and running. For the most part, installing Baïkal 0.7.0 was a straightforward process, but there were a couple of “gotchas” along the way.

The first (and most confusing) problem came after the installation/initial configuration when I tried to access my newly-created user’s calendar via the URL:

dav.MYDOMAIN.com/html/dav.php/calendar/MYUSERNAME/default

I knew that something was wrong when it wouldn’t even prompt me for credentials. Instead, the logs indicated the following error message:

[Tue Jun 02 14:13:05.529805 2020] [proxy_fcgi:error] [pid 32165:tid 139743908050688] [client 71.81.87.208:38910] AH01071: Got error 'PHP message: LogicException: Requested uri (/html/dav.php) is out of base uri (/s/html/dav.php/) in /var/www/domains/MYDOMAIN/dav/htdocs/vendor/sabre/http/lib/Request.php:184

I couldn’t figure out where the “/s/” was coming in before the “/html” portion, but that was certainly the cause of the error message. I filed an issue for it, and though I still don’t know the source of the problem, I was able to work around it by adding a trailing slash to the DocumentRoot for that particular vhost:

# pwd && diff -Nut dav.MYDOMAIN.conf.PRE-20200602_docroot dav.MYDOMAIN.conf
/etc/apache2/vhosts.d/includes
--- dav.MYDOMAIN.conf.PRE-20200602_docroot 2020-06-02 17:23:20.246281195 -0400 +++ dav.MYDOMAIN.conf 2020-06-02 17:20:59.892270352 -0400
@@ -1,7 +1,7 @@
- DocumentRoot "/var/www/domains/MYDOMAIN/dav/htdocs"
+ DocumentRoot "/var/www/domains/MYDOMAIN/dav/htdocs/"

After solving that strange problem, I was at least prompted for credentials when I accessed the calendar URL from above. After logging in, I ran into one more problem, though:

Class 'XMLWriter' not found

This problem was much easier to fix. I simply needed to add the ‘xmlwriter‘ USE flag to dev-lang/php (I also added ‘xmlreader‘ for good measure), emerge it again, and restart PHP-FPM. Other distributions (like CentOS) will likely need to install the ‘php-xml’ package (or something similar).

After that fix, I am happy to report that Baïkal 0.7.0 is working beautifully, and I have my calendars synced across all my devices. I personally use Thunderbird with Lightning on my computers, and a combination of DAVx5 with Simple Calendar Pro on my Android devices.

Just this past week, the new version of of Baïkal (0.7.0)—a PHP CalDAV and CardDAV server based on Sabre—was released, and one of the key changes was that support was added for more modern versions of PHP (like 7.4).

Since my personal Gentoo server is running the ~amd64 branch, I had to wait for this release in order to get my CalDAV server up and running. For the most part, installing Baïkal 0.7.0 was a straightforward process, but there were a couple of “gotchas” along the way.

The first (and most confusing) problem came after the installation/initial configuration when I tried to access my newly-created user’s calendar via the URL:

https://dav.MYDOMAIN.com/html/dav.php/calendar/MYUSERNAME/default

I knew that something was wrong when it wouldn’t even prompt me for credentials. Instead, the logs indicated the following error message:

[Tue Jun 02 14:13:05.529805 2020] [proxy_fcgi:error] [pid 32165:tid 139743908050688] [client 71.81.87.208:38910] AH01071: Got error 'PHP message: LogicException: Requested uri (/html/dav.php) is out of base uri (/s/html/dav.php/) in /var/www/domains/MYDOMAIN/dav/htdocs/vendor/sabre/http/lib/Request.php:184

I couldn’t figure out where the “/s/” was coming in before the “/html” portion, but that was certainly the cause of the error message. I filed an issue for it, and though I still don’t know the source of the problem, I was able to work around it by adding a trailing slash to the DocumentRoot for that particular vhost:

# pwd && diff -Nut dav.MYDOMAIN.conf.PRE-20200602_docroot dav.MYDOMAIN.conf
/etc/apache2/vhosts.d/includes
--- dav.MYDOMAIN.conf.PRE-20200602_docroot 2020-06-02 17:23:20.246281195 -0400 +++ dav.MYDOMAIN.conf 2020-06-02 17:20:59.892270352 -0400
@@ -1,7 +1,7 @@
- DocumentRoot "/var/www/domains/MYDOMAIN/dav/htdocs"
+ DocumentRoot "/var/www/domains/MYDOMAIN/dav/htdocs/"

After solving that strange problem, I was at least prompted for credentials when I accessed the calendar URL from above. After logging in, I ran into one more problem, though:

Class 'XMLWriter' not found

This problem was much easier to fix. I simply needed to add the ‘xmlwriter‘ USE flag to dev-lang/php (I also added ‘xmlreader‘ for good measure), emerge it again, and restart PHP-FPM. Other distributions (like CentOS) will likely need to install the ‘php-xml’ package (or something similar).

After that fix, I am happy to report that Baïkal 0.7.0 is working beautifully, and I have my calendars synced across all my devices. I personally use Thunderbird with Lightning on my computers, and a combination of DAVx5 with Simple Calendar Pro on my Android devices.

May 10 2020

200th Gentoo Council meeting

Gentoo News (GentooNews) May 10, 2020, 5:00

Way back in 2005, the reorganization of Gentoo led to the formation of the Gentoo Council, a steering body elected annually by the Gentoo developers. Forward 15 years, and today we had our 200th meeting! (No earth shaking decisions were taken today though.) The logs and summaries of all meetings can be read online on the archive page.

council group photo

Way back in 2005, the reorganization of Gentoo led to the formation of the Gentoo Council, a steering body elected annually by the Gentoo developers. Forward 15 years, and today we had our 200th meeting! (No earth shaking decisions were taken today though.) The logs and summaries of all meetings can be read online on the archive page.

May 08 2020

Reviving Gentoo Bugday

Gentoo News (GentooNews) May 08, 2020, 5:00

Reviving an old tradition, the next Gentoo Bugday will take place on Saturday 2020-06-06. Let’s contribute to Gentoo and fix bugs! We will focus on two topics in particular:

  • Adding or improving documentation on the Gentoo wiki
  • Fixing packages that fail with -fno-common (bug #705764)

Join us on channel #gentoo-bugday, freenode IRC, for real-time help. See you on 2020-06-06!

bug outline

Reviving an old tradition, the next Gentoo Bugday will take place on Saturday 2020-06-06. Let’s contribute to Gentoo and fix bugs! We will focus on two topics in particular:

  • Adding or improving documentation on the Gentoo wiki
  • Fixing packages that fail with -fno-common (bug #705764)

Join us on channel #gentoo-bugday, freenode IRC, for real-time help. See you on 2020-06-06!

April 16 2020

Why I stopped fuzzing research

Agostino Sarubbo (ago) April 16, 2020, 17:53

If you followed me in the past, you may have noticed that I stopped fuzzing research. During this time many people have asked me why…so instead of repeating the same answer every time, why not write a few lines about it…

While fuzz research was in my case fully automated, if you want to do a nice job you should:
– Communicate with upstream by making an exhaustive bug-report;
– Publish an advisory that collects all the needed info (affected versions, fixed version, commit fix, reproducer, poc, and so on) otherwise you force each downstream maintainer to do that by himself.

What happens in the majority of cases instead?
– When there is no ticketing system, upstream maintainers do not answer to your emails but fix the issues silently so, if you aren’t familiar with the code or if you don’t have time for investigations, you don’t have enough data to post. Even if you had time and you knew the code, you could still make a mistake; so why take the responsibility of pointing out commit fixes and so on?
– If you pass the above step, you have to request a CVE. In the past it was enough to publish on oss-security and you would get a CVE from a member of the Mitre team. Nowadays you have to fill a request that includes all the mentioned data and………wait ♦

If you pass the above two points and publish your advisory, what’s the next step? Stay tuned and wait for duplicates ♦ .

Let’s see a real example:
In the past I did fuzzing research on audiofile. Here is a screenshot of the issues without any words in the search field:

Do you see anything strange? Yeah there is clearly a duplicate.
I’m showing this image to point out the fact that, in order to avoid the duplicate, it would have been enough to look a little further below, so I am wondering:
if you are able to compile the software, use ASAN, use AFL, why aren’t you able to make a simple search to check if this issue was already filed?
For now, the only answer that I can think of is: everyone is hungry to find security issues and be the discoverer of a CVE.
Let’s clarify: if you find security issues by fuzzing you are not a security researcher at all and you will not be more palatable to the cybersecurity world. You are just creating CVE confusion for the rest of us.

On the other side, dear Mitre: you force us to fill an exhaustive request so, since you have all the data, why are you mistakenly assigning CVEs for already reported issues?

The first few times I saw these duplicates, I tried to report them but, unfortunately, it’s not my job and I found it very hard to do because of the large amount.

So, in short, I stopped fuzzing research because due to the current state of things, it’s a big waste of time.

If you followed me in the past, you may have noticed that I stopped fuzzing research. During this time many people have asked me why…so instead of repeating the same answer every time, why not write a few lines about it…

While fuzz research was in my case fully automated, if you want to do a nice job you should:
– Communicate with upstream by making an exhaustive bug-report;
– Publish an advisory that collects all the needed info (affected versions, fixed version, commit fix, reproducer, poc, and so on) otherwise you force each downstream maintainer to do that by himself.

What happens in the majority of cases instead?
– When there is no ticketing system, upstream maintainers do not answer to your emails but fix the issues silently so, if you aren’t familiar with the code or if you don’t have time for investigations, you don’t have enough data to post. Even if you had time and you knew the code, you could still make a mistake; so why take the responsibility of pointing out commit fixes and so on?
– If you pass the above step, you have to request a CVE. In the past it was enough to publish on oss-security and you would get a CVE from a member of the Mitre team. Nowadays you have to fill a request that includes all the mentioned data and………wait 😀

If you pass the above two points and publish your advisory, what’s the next step? Stay tuned and wait for duplicates 😀 .

Let’s see a real example:
In the past I did fuzzing research on audiofile. Here is a screenshot of the issues without any words in the search field:

Do you see anything strange? Yeah there is clearly a duplicate.
I’m showing this image to point out the fact that, in order to avoid the duplicate, it would have been enough to look a little further below, so I am wondering:
if you are able to compile the software, use ASAN, use AFL, why aren’t you able to make a simple search to check if this issue was already filed?
For now, the only answer that I can think of is: everyone is hungry to find security issues and be the discoverer of a CVE.
Let’s clarify: if you find security issues by fuzzing you are not a security researcher at all and you will not be more palatable to the cybersecurity world. You are just creating CVE confusion for the rest of us.

On the other side, dear Mitre: you force us to fill an exhaustive request so, since you have all the data, why are you mistakenly assigning CVEs for already reported issues?

The first few times I saw these duplicates, I tried to report them but, unfortunately, it’s not my job and I found it very hard to do because of the large amount.

So, in short, I stopped fuzzing research because due to the current state of things, it’s a big waste of time.

April 15 2020

Spam increase due to SpamAssassin Bayes database not available for scanning

Nathan Zachary (nathanzachary) April 15, 2020, 20:04

For approximately the past six weeks or so, I’ve noticed an uptick in the amount of spam getting through (and delivered) on my primary mail server. At first the increase in false negatives (meaning spam not getting flagged as such) wasn’t all that bad, so I didn’t think much of it. However, starting last week and especially this week, the increase was so dramatic that it prompted me to look further into the problem.

I started by looking through my SpamAssassin and amavis settings to make sure that nothing was blatantly wrong, but nothing stood out as having recently changed. I made sure that I had the required Perl modules for all of SpamAssassin’s filtering, and again, nothing had recently changed. Coming up empty-handed, I decided to take a look at some headers for an email that came through even though it was spam:

X-Spam-Status: No, score=1.502 required=3.2 tests=[
	DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.25,
	FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25,
	HTML_MESSAGE=0.001]

I thought that possibly something had changed with how SpamAssassin assigned points for various tests, and temporarily dropped the score required for spam flagging to 1.4. Delving more deeply, though, I found that the point assignments had not changed, so I reverted to 3.2 and kept investigating. After looking again, I noticed that one key test wasn’t showing in the ‘X-Spam-Status’ header for this email: Bayesian filtering. Normally, there would be some type of reference to ‘BAYES_%=#’ (where % represents the chance that the Bayesian filter thought that the message could be spam and # represents the score assigned to the email based on that chance) in the spam header. However, it was no longer showing up, which indicated to me that the Bayes filters weren’t running.

I then started with some basic Bayes troubleshooting steps, and found some clues. By analysing the output of spamassassin -D --lint, I saw that there could be problem with the Bayes database:

Apr 14 22:10:41.875 [20701] dbg: plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC
Apr 14 22:10:42.061 [20701] dbg: config: fixed relative path: /var/lib/spamassassin/3.004004/updates_spamassassin_org/23_bayes.cf
Apr 14 22:10:42.061 [20701] dbg: config: using "/var/lib/spamassassin/3.004004/updates_spamassassin_org/23_bayes.cf" for included file
Apr 14 22:10:42.061 [20701] dbg: config: read file /var/lib/spamassassin/3.004004/updates_spamassassin_org/23_bayes.cf
Apr 14 22:10:42.748 [20701] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55f1700749b8) implements 'learner_new', priority 0
Apr 14 22:10:42.748 [20701] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x55f1700749b8), bayes_store_module=Mail::SpamAssassin::BayesStore::DBM
Apr 14 22:10:42.759 [20701] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::DBM=HASH(0x55f170db5ba8)
Apr 14 22:10:42.759 [20701] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55f1700749b8) implements 'learner_is_scan_available', priority 0
Apr 14 22:10:42.759 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Apr 14 22:10:42.760 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen
Apr 14 22:10:42.760 [20701] dbg: bayes: found bayes db version 3
Apr 14 22:10:42.760 [20701] dbg: bayes: DB journal sync: last sync: 1586894330
Apr 14 22:10:42.760 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200
Apr 14 22:10:42.760 [20701] dbg: bayes: untie-ing
Apr 14 22:10:42.764 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Apr 14 22:10:42.764 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen
Apr 14 22:10:42.765 [20701] dbg: bayes: found bayes db version 3
Apr 14 22:10:42.765 [20701] dbg: bayes: DB journal sync: last sync: 1586894330
Apr 14 22:10:42.765 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200
Apr 14 22:10:42.765 [20701] dbg: bayes: untie-ing

In particular, the following lines indicated a problem to me:

Apr 14 22:10:42.760 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200
<snip>
Apr 14 22:10:42.765 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200

Thinking back, I remembered that there were some changes to the amavis implementation in Gentoo that caused me problems in late February of 2020. One of those changes was relocating the amavis user’s home/runtime directory from /var/amavis/ to /var/lib/amavishome/. That’s when I saw it in the debugging output:

Apr 14 22:10:42.759 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Apr 14 22:10:42.760 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen

The directory for the Bayes database shouldn’t be /var/amavis/.spamassassin/bayes* any longer, but instead should be /var/lib/amavishome/.spamassassin/bayes*. I made that change:

# grep bayes_path /etc/mail/spamassassin/local.cf 
bayes_path /var/lib/amavishome/.spamassassin/bayes

and restarted both amavis and spamd, and now I could see the Bayes filter in the ‘X-Spam-Status’ header:

X-Spam-Status: Yes, score=18.483 required=3.2 tests=[BAYES_99=5.75,
	BAYES_999=8, FROM_SUSPICIOUS_NTLD=0.499,
	FROM_SUSPICIOUS_NTLD_FP=0.514, HTML_MESSAGE=0.001,
	HTML_OFF_PAGE=0.927, PDS_OTHER_BAD_TLD=1.999, RDNS_NONE=0.793]

After implementing the change for the Bayes database location in amavis, I have seen the false negative level drop back to where it used to be. ♦

Cheers,
Zach

For approximately the past six weeks or so, I’ve noticed an uptick in the amount of spam getting through (and delivered) on my primary mail server. At first the increase in false negatives (meaning spam not getting flagged as such) wasn’t all that bad, so I didn’t think much of it. However, starting last week and especially this week, the increase was so dramatic that it prompted me to look further into the problem.

I started by looking through my SpamAssassin and amavis settings to make sure that nothing was blatantly wrong, but nothing stood out as having recently changed. I made sure that I had the required Perl modules for all of SpamAssassin’s filtering, and again, nothing had recently changed. Coming up empty-handed, I decided to take a look at some headers for an email that came through even though it was spam:

X-Spam-Status: No, score=1.502 required=3.2 tests=[
	DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.25,
	FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25,
	HTML_MESSAGE=0.001]

I thought that possibly something had changed with how SpamAssassin assigned points for various tests, and temporarily dropped the score required for spam flagging to 1.4. Delving more deeply, though, I found that the point assignments had not changed, so I reverted to 3.2 and kept investigating. After looking again, I noticed that one key test wasn’t showing in the ‘X-Spam-Status’ header for this email: Bayesian filtering. Normally, there would be some type of reference to ‘BAYES_%=#’ (where % represents the chance that the Bayesian filter thought that the message could be spam and # represents the score assigned to the email based on that chance) in the spam header. However, it was no longer showing up, which indicated to me that the Bayes filters weren’t running.

I then started with some basic Bayes troubleshooting steps, and found some clues. By analysing the output of spamassassin -D --lint, I saw that there could be problem with the Bayes database:

Apr 14 22:10:41.875 [20701] dbg: plugin: loading Mail::SpamAssassin::Plugin::Bayes from @INC
Apr 14 22:10:42.061 [20701] dbg: config: fixed relative path: /var/lib/spamassassin/3.004004/updates_spamassassin_org/23_bayes.cf
Apr 14 22:10:42.061 [20701] dbg: config: using "/var/lib/spamassassin/3.004004/updates_spamassassin_org/23_bayes.cf" for included file
Apr 14 22:10:42.061 [20701] dbg: config: read file /var/lib/spamassassin/3.004004/updates_spamassassin_org/23_bayes.cf
Apr 14 22:10:42.748 [20701] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55f1700749b8) implements 'learner_new', priority 0
Apr 14 22:10:42.748 [20701] dbg: bayes: learner_new self=Mail::SpamAssassin::Plugin::Bayes=HASH(0x55f1700749b8), bayes_store_module=Mail::SpamAssassin::BayesStore::DBM
Apr 14 22:10:42.759 [20701] dbg: bayes: learner_new: got store=Mail::SpamAssassin::BayesStore::DBM=HASH(0x55f170db5ba8)
Apr 14 22:10:42.759 [20701] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x55f1700749b8) implements 'learner_is_scan_available', priority 0
Apr 14 22:10:42.759 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Apr 14 22:10:42.760 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen
Apr 14 22:10:42.760 [20701] dbg: bayes: found bayes db version 3
Apr 14 22:10:42.760 [20701] dbg: bayes: DB journal sync: last sync: 1586894330
Apr 14 22:10:42.760 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200
Apr 14 22:10:42.760 [20701] dbg: bayes: untie-ing
Apr 14 22:10:42.764 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Apr 14 22:10:42.764 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen
Apr 14 22:10:42.765 [20701] dbg: bayes: found bayes db version 3
Apr 14 22:10:42.765 [20701] dbg: bayes: DB journal sync: last sync: 1586894330
Apr 14 22:10:42.765 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200
Apr 14 22:10:42.765 [20701] dbg: bayes: untie-ing

In particular, the following lines indicated a problem to me:

Apr 14 22:10:42.760 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200
<snip>
Apr 14 22:10:42.765 [20701] dbg: bayes: not available for scanning, only 0 ham(s) in bayes DB < 200

Thinking back, I remembered that there were some changes to the amavis implementation in Gentoo that caused me problems in late February of 2020. One of those changes was relocating the amavis user’s home/runtime directory from /var/amavis/ to /var/lib/amavishome/. That’s when I saw it in the debugging output:

Apr 14 22:10:42.759 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_toks
Apr 14 22:10:42.760 [20701] dbg: bayes: tie-ing to DB file R/O /var/amavis/.spamassassin/bayes_seen

The directory for the Bayes database shouldn’t be /var/amavis/.spamassassin/bayes* any longer, but instead should be /var/lib/amavishome/.spamassassin/bayes*. I made that change:

# grep bayes_path /etc/mail/spamassassin/local.cf 
bayes_path /var/lib/amavishome/.spamassassin/bayes

and restarted both amavis and spamd, and now I could see the Bayes filter in the ‘X-Spam-Status’ header:

X-Spam-Status: Yes, score=18.483 required=3.2 tests=[BAYES_99=5.75,
	BAYES_999=8, FROM_SUSPICIOUS_NTLD=0.499,
	FROM_SUSPICIOUS_NTLD_FP=0.514, HTML_MESSAGE=0.001,
	HTML_OFF_PAGE=0.927, PDS_OTHER_BAD_TLD=1.999, RDNS_NONE=0.793]

After implementing the change for the Bayes database location in amavis, I have seen the false negative level drop back to where it used to be. 🙂

Cheers,
Zach

April 14 2020

py3status v3.28 – goodbye py2.6-3.4

Alexys Jacob (ultrabug) April 14, 2020, 14:51

The newest version of py3status starts to enforce the deprecation of Python 2.6 to 3.4 (included) initiated by Thiago Kenji Okada more than a year ago and orchestrated by Hugo van Kemenade via #1904 and #1896.

Thanks to Hugo, I discovered a nice tool by @asottile to update your Python code base to recent syntax sugars called pyupgrade!

Debian buster users might be interested in the installation war story that @TRS-80 kindly described and the final (and documented) solution found.

Changelog since v3.26
  • drop support for EOL Python 2.6-3.4 (#1896), by Hugo van Kemenade
  • i3status: support read_file module (#1909), by @lasers thx to @dohseven
  • clock module: add “locale” config parameter to change time representation (#1910), by inemajo
  • docs: update debian instructions fix #1916
  • mpd_status module: use currentsong command if possible (#1924), by girst
  • networkmanager module: allow using the currently active AP in formats (#1921), by Benoît Dardenne
  • volume_status module: change amixer flag ordering fix #1914 (#1920)
Thank you contributors
  • Thiago Kenji Okada
  • Hugo van Kemenade
  • Benoît Dardenne
  • @dohseven
  • @inemajo
  • @girst
  • @lasers

The newest version of py3status starts to enforce the deprecation of Python 2.6 to 3.4 (included) initiated by Thiago Kenji Okada more than a year ago and orchestrated by Hugo van Kemenade via #1904 and #1896.

Thanks to Hugo, I discovered a nice tool by @asottile to update your Python code base to recent syntax sugars called pyupgrade!

Debian buster users might be interested in the installation war story that @TRS-80 kindly described and the final (and documented) solution found.

Changelog since v3.26

  • drop support for EOL Python 2.6-3.4 (#1896), by Hugo van Kemenade
  • i3status: support read_file module (#1909), by @lasers thx to @dohseven
  • clock module: add “locale” config parameter to change time representation (#1910), by inemajo
  • docs: update debian instructions fix #1916
  • mpd_status module: use currentsong command if possible (#1924), by girst
  • networkmanager module: allow using the currently active AP in formats (#1921), by Benoît Dardenne
  • volume_status module: change amixer flag ordering fix #1914 (#1920)

Thank you contributors

  • Thiago Kenji Okada
  • Hugo van Kemenade
  • Benoît Dardenne
  • @dohseven
  • @inemajo
  • @girst
  • @lasers

Zstandard (zstd) Coming to >= gentoo-sources-5.6.4 (use=experimental)

Mike Pagano (mpagano) April 08, 2020, 18:15

I just added zstd to gentoo-sources which will apply to gentoo-sources kernels >=5.6.4 when the ‘experimental’ use flag is enabled.

zstd is described here[1] as “…a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. It’s backed by a very fast entropy stage, provided by Huff0 and FSE library.”

You can read more about it here[2].

Thanks to Klemen Mihevc for the request [3]

[1] github.com/facebook/zstd

[2]facebook.github.io/zstd/

[3] bugs.gentoo.org/716520

I just added zstd to gentoo-sources which will apply to gentoo-sources kernels >=5.6.4 when the ‘experimental’ use flag is enabled.

zstd is described here[1] as “…a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. It’s backed by a very fast entropy stage, provided by Huff0 and FSE library.”

You can read more about it here[2].

Thanks to Klemen Mihevc for the request [3]

[1] https://github.com/facebook/zstd

[2]https://facebook.github.io/zstd/

[3] https://bugs.gentoo.org/716520

March 30 2020

Linux kernel 5.6.0 iwlwifi bug

Mike Pagano (mpagano) March 30, 2020, 13:27

Quick note that the Linux kernel 5.6.0 has an iwlwifi bug that will prevent network connectivity. [1]

A patch is out but did not make 5.6.0. This patch IS included in gentoo-sources-5.6.0. It will be in a future vanilla-sources 5.6.X once upstream releases a new version.

[1] www.phoronix.com/scan.php?page=news_item&px=Linux-5.6-Broken-Intel-IWLWIFI

[2] git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/patch/?id=be8c827f50a0bcd56361b31ada11dc0a3c2fd240

Quick note that the Linux kernel 5.6.0 has an iwlwifi bug that will prevent network connectivity. [1]

A patch is out but did not make 5.6.0. This patch IS included in gentoo-sources-5.6.0. It will be in a future vanilla-sources 5.6.X once upstream releases a new version.

[1] https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.6-Broken-Intel-IWLWIFI

[2] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/patch/?id=be8c827f50a0bcd56361b31ada11dc0a3c2fd240

February 27 2020

Amavisd crashes immediately on start with little in the logs

Nathan Zachary (nathanzachary) February 27, 2020, 17:02

Recently I was updating amavisd and other portions of the mail stack on one of my mail servers (running Gentoo Linux). This particular set of updates didn’t go as smoothly as I would have liked primarily due to the acct-user/amavis package being created on Thursday, 13 February 2020 as a part of the ongoing effort to standardise user accounts used within applications. Some other problems with the upgrade of amavisd (to version 2.12.0-r1) and the mail stack were that some dependencies were erroneously removed from the ebuild (bug 710842) and (though not directly related) permissions within the new amavis user’s home directory being inadequate for ClamAV to successfully run. I will address these problems and their respective fixes below.

Missing dependencies in 2.12.0-r1 (bug 710842)

Some of the Perl modules (since amavisd-new is written in Perl) were erroneously removed from the dependency list in version 2.12.0-r1. The fix here was to simply reinstall the modules manually, which in my case entailed:

emerge -av MIME-tools Net-Server Mail-DKIM MailTools Net-LibIDN

However, my situation required one additional module that wasn’t mentioned in the bug, and was discovered when trying to manually start amavisd because of this error message:

# /usr/sbin/amavisd
Problem in Amavis::Unpackers code: Can't locate Archive/Zip.pm in @INC (you may need to install the Archive::Zip module) (@INC contains: /etc/perl /usr/local/lib64/perl5/5.30.1/x86_64-linux /usr/local/lib64/perl5/5.30.1 /usr/lib64/perl5/vendor_perl/5.30.1/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.1 /usr/local/lib64/perl5 /usr/lib64/perl5/vendor_perl/5.30.0/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.0 /usr/lib64/perl5/vendor_perl/5.28.0 /usr/lib64/perl5/vendor_perl/5.26.2 /usr/lib64/perl5/vendor_perl/5.26.1 /usr/lib64/perl5/vendor_perl/5.24.0 /usr/lib64/perl5/vendor_perl /usr/lib64/perl5/5.30.1/x86_64-linux /usr/lib64/perl5/5.30.1) at (eval 101) line 47.
BEGIN failed--compilation aborted at (eval 101) line 47.

That one was also simple enough to solve by installing dev-perl/Archive-Zip:

emerge -av dev-perl/Archive-Zip

Fortunately, this ‘missing dependencies’ problem was fixed with version 2.12.0-r2.

Changes in the ‘amavis’ user’s home directory

With the addition of the acct-user/amavis package, a few things changes about the default setup for the ‘amavis’ user. When installing that package, Portage warns about some steps that need to be implemented in order to make amavis work again after migrating its home directory from /var/amavis/ to /var/lib/amavishome. Even after following those steps, I saw the following error message when starting amavis:

Feb 27 00:08:31 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:08:31 [amavis] will bind to /var/amavis/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Amavis (type Net::Server::PreForkSimple) starting! pid(1738)
Feb 27 00:08:31 [amavis] Net::Server: Binding to UNIX socket file "/var/amavis/amavisd.sock"
Feb 27 00:08:31 [amavis] (!)Net::Server: 2020/02/27-00:08:31 Can't connect to UNIX socket at file /var/amavis/amavisd.sock [No such file or directory]\n  at line 66 in file /usr/lib64/perl5/vendor_perl/5.30.1/Net/Server/Proto/UNIX.pm
Feb 27 00:08:31 [amavis] sd_notify (no socket): STOPPING=1\nSTATUS=Server rundown, notifying child processes.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Server closing!
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Child processes have been stopped.

In that error message, I noticed that there were still references to /var/amavis/ instead of the new /var/lib/amavishome/ directory, so I updated it using the $MYHOME variable in /etc/amavisd.conf:

# grep -e '^$MYHOME' /etc/amavisd.conf 
$MYHOME = '/var/lib/amavishome';   # a convenient default for other settings, -H

Thereafter, the start-up logs indicated that it was binding to a UNIX socket in the correct home directory:

Feb 27 00:12:15 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:12:15 [amavis] will bind to /var/lib/amavishome/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:12:15 [amavis] Net::Server: 2020/02/27-00:12:15 Amavis (type Net::Server::PreForkSimple) starting! pid(1952)
Feb 27 00:12:15 [amavis] Net::Server: Binding to UNIX socket file "/var/lib/amavishome/amavisd.sock"
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host 127.0.0.1 with IPv4
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host ::1 with IPv6
ClamAV permissions within the amavis home directory:

Though this error wasn’t directly related to the upgrades (and had likely existed for quite some time beforehand), I only just now noticed it whilst combing through the logs (the first error is from my system’s mail log, and the second error is from the clamd log):

Feb 27 00:15:10 [amavis] (01980-01) run_av (ClamAV-clamd) result: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n
Feb 27 00:15:10 [amavis] (01980-01) (!)run_av (ClamAV-clamd) FAILED - unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n"
Feb 27 00:15:10 [amavis] (01980-01) (!)ClamAV-clamd av-scanner FAILED: CODE(0x5611198fa5d8) unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n" at (eval 85) line 951.
# grep 'lstat' /var/log/clamav/clamd.log
Wed Feb 26 23:26:53 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180007-00529-S1gbY8cd/parts
Wed Feb 26 23:28:32 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180038-00592-FK2_Uj2T/parts
Wed Feb 26 23:31:30 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T165427-32346-V4WeP0YX/parts
Thu Feb 27 00:15:10 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts
Thu Feb 27 00:26:21 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002621-01981-l36mWT4P/parts
Thu Feb 27 00:26:30 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002630-01982-k0qgJdjl/parts
Thu Feb 27 00:29:12 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002912-01983-G57aKCmK/parts
Thu Feb 27 00:31:55 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003155-01984-M9r9r1Gc/parts
Thu Feb 27 00:33:07 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003307-01985-8n_wS6pQ/parts
Thu Feb 27 00:40:20 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004020-01986-gH4PrFAk/parts
Thu Feb 27 00:45:45 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004545-01987-J4YGuNbw/parts

These errors were fixed by 1) adding the ‘clamav’ user to the ‘amavis’ group, 2) setting g+w permissions on the /var/lib/amavishome/tmp/ directory, and 3) restarting clamd and amavisd

# gpasswd -a clamav amavis
Adding user clamav to group amavis

# ls -alh /var/lib/amavishome/ | grep tmp
drwxr-xr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# chmod 775 /var/lib/amavishome/tmp

# ls -alh /var/lib/amavishome/ | grep tmp
drwxrwxr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# /etc/init.d/clamd restart && /etc/init.d/amavisd restart

Now ClamAV is readily able to access files under /var/lib/amavishome/tmp/:

Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Connecting to socket  /var/run/clamav/clamd.sock
Feb 27 01:00:12 [amavis] (03346-01) new socket by IO::Socket::UNIX to /var/run/clamav/clamd.sock, timeout set to 10
Feb 27 01:00:12 [amavis] (03346-01) connected to /var/run/clamav/clamd.sock successfully
Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Sending CONTSCAN /var/lib/amavishome/tmp/amavis-20200227T010012-03346-YjbUpYOk/parts\n to socket /var/run/clamav/clamd.sock

Hopefully, if you run into these errors, you will be able to take this information and apply it to your particular mail stack.

Cheers,

Zach

Recently I was updating amavisd and other portions of the mail stack on one of my mail servers (running Gentoo Linux). This particular set of updates didn’t go as smoothly as I would have liked primarily due to the acct-user/amavis package being created on Thursday, 13 February 2020 as a part of the ongoing effort to standardise user accounts used within applications. Some other problems with the upgrade of amavisd (to version 2.12.0-r1) and the mail stack were that some dependencies were erroneously removed from the ebuild (bug 710842) and (though not directly related) permissions within the new amavis user’s home directory being inadequate for ClamAV to successfully run. I will address these problems and their respective fixes below.

Missing dependencies in 2.12.0-r1 (bug 710842)

Some of the Perl modules (since amavisd-new is written in Perl) were erroneously removed from the dependency list in version 2.12.0-r1. The fix here was to simply reinstall the modules manually, which in my case entailed:

emerge -av MIME-tools Net-Server Mail-DKIM MailTools Net-LibIDN

However, my situation required one additional module that wasn’t mentioned in the bug, and was discovered when trying to manually start amavisd because of this error message:

# /usr/sbin/amavisd
Problem in Amavis::Unpackers code: Can't locate Archive/Zip.pm in @INC (you may need to install the Archive::Zip module) (@INC contains: /etc/perl /usr/local/lib64/perl5/5.30.1/x86_64-linux /usr/local/lib64/perl5/5.30.1 /usr/lib64/perl5/vendor_perl/5.30.1/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.1 /usr/local/lib64/perl5 /usr/lib64/perl5/vendor_perl/5.30.0/x86_64-linux /usr/lib64/perl5/vendor_perl/5.30.0 /usr/lib64/perl5/vendor_perl/5.28.0 /usr/lib64/perl5/vendor_perl/5.26.2 /usr/lib64/perl5/vendor_perl/5.26.1 /usr/lib64/perl5/vendor_perl/5.24.0 /usr/lib64/perl5/vendor_perl /usr/lib64/perl5/5.30.1/x86_64-linux /usr/lib64/perl5/5.30.1) at (eval 101) line 47.
BEGIN failed--compilation aborted at (eval 101) line 47.

That one was also simple enough to solve by installing dev-perl/Archive-Zip:

emerge -av dev-perl/Archive-Zip

Fortunately, this ‘missing dependencies’ problem was fixed with version 2.12.0-r2.

Changes in the ‘amavis’ user’s home directory

With the addition of the acct-user/amavis package, a few things changes about the default setup for the ‘amavis’ user. When installing that package, Portage warns about some steps that need to be implemented in order to make amavis work again after migrating its home directory from /var/amavis/ to /var/lib/amavishome. Even after following those steps, I saw the following error message when starting amavis:

Feb 27 00:08:31 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:08:31 [amavis] will bind to /var/amavis/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Amavis (type Net::Server::PreForkSimple) starting! pid(1738)
Feb 27 00:08:31 [amavis] Net::Server: Binding to UNIX socket file "/var/amavis/amavisd.sock"
Feb 27 00:08:31 [amavis] (!)Net::Server: 2020/02/27-00:08:31 Can't connect to UNIX socket at file /var/amavis/amavisd.sock [No such file or directory]\n  at line 66 in file /usr/lib64/perl5/vendor_perl/5.30.1/Net/Server/Proto/UNIX.pm
Feb 27 00:08:31 [amavis] sd_notify (no socket): STOPPING=1\nSTATUS=Server rundown, notifying child processes.
Feb 27 00:08:31 [amavis] Net::Server: 2020/02/27-00:08:31 Server closing!
Feb 27 00:08:31 [amavis] sd_notify (no socket): STATUS=Child processes have been stopped.

In that error message, I noticed that there were still references to /var/amavis/ instead of the new /var/lib/amavishome/ directory, so I updated it using the $MYHOME variable in /etc/amavisd.conf:

# grep -e '^$MYHOME' /etc/amavisd.conf 
$MYHOME = '/var/lib/amavishome';   # a convenient default for other settings, -H

Thereafter, the start-up logs indicated that it was binding to a UNIX socket in the correct home directory:

Feb 27 00:12:15 [amavis] socket module IO::Socket::IP, protocol families available: INET, INET6
Feb 27 00:12:15 [amavis] will bind to /var/lib/amavishome/amavisd.sock|unix, 127.0.0.1:10024/tcp, [::1]:10024/tcp
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Transferring control to Net::Server.
Feb 27 00:12:15 [amavis] sd_notify (no socket): STATUS=Preparing to bind sockets.
Feb 27 00:12:15 [amavis] Net::Server: 2020/02/27-00:12:15 Amavis (type Net::Server::PreForkSimple) starting! pid(1952)
Feb 27 00:12:15 [amavis] Net::Server: Binding to UNIX socket file "/var/lib/amavishome/amavisd.sock"
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host 127.0.0.1 with IPv4
Feb 27 00:12:15 [amavis] Net::Server: Binding to TCP port 10024 on host ::1 with IPv6

ClamAV permissions within the amavis home directory:

Though this error wasn’t directly related to the upgrades (and had likely existed for quite some time beforehand), I only just now noticed it whilst combing through the logs (the first error is from my system’s mail log, and the second error is from the clamd log):

Feb 27 00:15:10 [amavis] (01980-01) run_av (ClamAV-clamd) result: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n
Feb 27 00:15:10 [amavis] (01980-01) (!)run_av (ClamAV-clamd) FAILED - unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n"
Feb 27 00:15:10 [amavis] (01980-01) (!)ClamAV-clamd av-scanner FAILED: CODE(0x5611198fa5d8) unexpected , output="/var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts: lstat() failed: Permission denied. ERROR\n" at (eval 85) line 951.
# grep 'lstat' /var/log/clamav/clamd.log
Wed Feb 26 23:26:53 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180007-00529-S1gbY8cd/parts
Wed Feb 26 23:28:32 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T180038-00592-FK2_Uj2T/parts
Wed Feb 26 23:31:30 2020 -> WARNING: lstat() failed on: /var/amavis/tmp/amavis-20200226T165427-32346-V4WeP0YX/parts
Thu Feb 27 00:15:10 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T001510-01980-zrKbp28h/parts
Thu Feb 27 00:26:21 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002621-01981-l36mWT4P/parts
Thu Feb 27 00:26:30 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002630-01982-k0qgJdjl/parts
Thu Feb 27 00:29:12 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T002912-01983-G57aKCmK/parts
Thu Feb 27 00:31:55 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003155-01984-M9r9r1Gc/parts
Thu Feb 27 00:33:07 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T003307-01985-8n_wS6pQ/parts
Thu Feb 27 00:40:20 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004020-01986-gH4PrFAk/parts
Thu Feb 27 00:45:45 2020 -> WARNING: lstat() failed on: /var/lib/amavishome/tmp/amavis-20200227T004545-01987-J4YGuNbw/parts

These errors were fixed by 1) adding the ‘clamav’ user to the ‘amavis’ group, 2) setting g+w permissions on the /var/lib/amavishome/tmp/ directory, and 3) restarting clamd and amavisd

# gpasswd -a clamav amavis
Adding user clamav to group amavis

# ls -alh /var/lib/amavishome/ | grep tmp
drwxr-xr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# chmod 775 /var/lib/amavishome/tmp

# ls -alh /var/lib/amavishome/ | grep tmp
drwxrwxr-x 141 amavis amavis  12K Feb 27 00:46 tmp

# /etc/init.d/clamd restart && /etc/init.d/amavisd restart

Now ClamAV is readily able to access files under /var/lib/amavishome/tmp/:

Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Connecting to socket  /var/run/clamav/clamd.sock
Feb 27 01:00:12 [amavis] (03346-01) new socket by IO::Socket::UNIX to /var/run/clamav/clamd.sock, timeout set to 10
Feb 27 01:00:12 [amavis] (03346-01) connected to /var/run/clamav/clamd.sock successfully
Feb 27 01:00:12 [amavis] (03346-01) ClamAV-clamd: Sending CONTSCAN /var/lib/amavishome/tmp/amavis-20200227T010012-03346-YjbUpYOk/parts\n to socket /var/run/clamav/clamd.sock

Hopefully, if you run into these errors, you will be able to take this information and apply it to your particular mail stack.

Cheers,

Zach

February 23 2020

Searx and Gentoo wiki search

Alice Ferrazzi (alicef) February 23, 2020, 15:00

Two years ago I started to get interested in selfhosting services. I started to go away from private services and implementing selfhosting, manly because private services was disabling most of the features that I liked and I had no way to contribute or see how they was working.
That is what made look into old.reddit.com/r/selfhosted/ and www.privacytools.io/ That is when I disovered searx, as the github page say searx is a "Privacy-respecting metasearch engine".
As any selfhost service, you can install easily on your server or
also on the local computer.
For the installation instruction go here or use the searx-docker project.
As I use selfhosted services also because I like to contribute back,
after a few look I decided to add a meta-engine in searx.
Specifically a Gentoo wiki search meta-engine. Pull requeset #1368

Gentoo wiki search is usually enabled by default in the
it tab :)

Gentoo wiki search can also be used by the searx shortcut system (same as bang in duckduckgo if you have familiarity)

The Gentoo wiki search shortcut is !ge
for example
search.stinpriza.org/?q=%21ge%20project%3Akernel&categories=none&language=en

will give you this:

for concluding, have fun with searx and Gentoo!

I also highly recommend to have your own searx instance but you can play with public instances.

Two years ago I started to get interested in selfhosting services. I started to go away from private services and implementing selfhosting, manly because private services was disabling most of the features that I liked and I had no way to contribute or see how they was working.
That is what made look into https://old.reddit.com/r/selfhosted/ and https://www.privacytools.io/ That is when I disovered searx, as the github page say searx is a "Privacy-respecting metasearch engine".
As any selfhost service, you can install easily on your server or
also on the local computer.
For the installation instruction go here or use the searx-docker project.
As I use selfhosted services also because I like to contribute back,
after a few look I decided to add a meta-engine in searx.
Specifically a Gentoo wiki search meta-engine. Pull requeset #1368

Gentoo wiki search is usually enabled by default in the
it tab :)

searx_gentoo_default

Gentoo wiki search can also be used by the searx shortcut system (same as bang in duckduckgo if you have familiarity)

The Gentoo wiki search shortcut is !ge
for example
https://search.stinpriza.org/?q=%21ge%20project%3Akernel&categories=none&language=en

will give you this:
searx_shortcut_ge

for concluding, have fun with searx and Gentoo!

I also highly recommend to have your own searx instance but you can play with public instances.

February 21 2020

Gentoo Python Guide

Michał Górny (mgorny) February 21, 2020, 10:35

Gentoo provides one of the best frameworks for providing Python support in packages among operating systems. This includes support for running multiple versions of Python (while most other distributions avoid going beyond simultaneous support for Python 2 and one version of Python 3), alternative implementations of Python, reliable tests, deep QA checks. While we aim to keep things simple, this is not always possible.

At the same time, the available documentation is limited and not always up-to-date. Both the built-in eclass documentation and Python project wiki page provide bits of documentation but they are mostly in reference form and not very suitable for beginners nor people who do not actively follow the developments within the ecosystem. This results in suboptimal ebuilds, improper dependencies, missing tests.

Gentoo Python Guide aims to fill the gap by providing a good, complete, by-topic (rather than reference-style) documentation for the ecosystem in Gentoo and the relevant eclasses. Combined with examples, it should help you write good ebuilds and solve common problems as simply as possible.

Gentoo Python Guide sources are available on GitHub. Suggestions and improvements are welcome.

mgorny (mgorny ) February 21, 2020, 10:35

Gentoo provides one of the best frameworks for providing Python support in packages among operating systems. This includes support for running multiple versions of Python (while most other distributions avoid going beyond simultaneous support for Python 2 and one version of Python 3), alternative implementations of Python, reliable tests, deep QA checks. While we aim to keep things simple, this is not always possible.

At the same time, the available documentation is limited and not always up-to-date. Both the built-in eclass documentation and Python project wiki page provide bits of documentation but they are mostly in reference form and not very suitable for beginners nor people who do not actively follow the developments within the ecosystem. This results in suboptimal ebuilds, improper dependencies, missing tests.

Gentoo Python Guide aims to fill the gap by providing a good, complete, by-topic (rather than reference-style) documentation for the ecosystem in Gentoo and the relevant eclasses. Combined with examples, it should help you write good ebuilds and solve common problems as simply as possible.

Gentoo Python Guide sources are available on GitHub. Suggestions and improvements are welcome.

February 10 2020

No more PYTHON_TARGETS in single-r1

Michał Górny (mgorny) February 10, 2020, 7:39

Since its inception in 2012, python-single-r1 has been haunting users with two sets of USE flags: PYTHON_TARGETS and PYTHON_SINGLE_TARGET. While this initially seemed a necessary part of the grand design, today I know we could have done better. Today this chymera is disappearing for real, and python-single-r1 are going to use PYTHON_SINGLE_TARGET flags only.

I would like to take this opportunity to explain why the eclass has been designed this way in the first place, and what has been done to change that.

Why PYTHON_SINGLE_TARGET?

Why did we need a second variable in the first place? After all, we could probably get away with using PYTHON_TARGETS everywhere, and adding an appropriate REQUIRED_USE constraint.

Back in the day we have established that for users’ convenience we need to default to enabling one version of Python 2 and one version of Python 3. If we enabled only one of them, the users would end up having to enable the other for a lot of packages. On the other had, if we combined both with using PT for single-r1 packages, the users would have to disable the extra implementation for a lot of them. Neither option was good.

The primary purpose of PYTHON_SINGLE_TARGET was to provide a parallel sensible setting for those packages. It was not only to make the default work out of the box but also to let users change it in one step.

Today, with the demise of Python 2 and the effort to remove Python 2 from default PT, it may seem less important to keep the distinction. Nevertheless, a number of developers and at least some users keep multiple versions of Python in PT to test their packages. Having PST is still helpful to them.

Why additional PYTHON_TARGETS then?

PST is only half of the story. What I explained above does not justify having PYTHON_TARGETS on those packages as well, and a REQUIRED_USE constraint to make them superset of enabled PST. Why did we need to have two flag sets then?

The answer is: PYTHON_USEDEP. The initial design goal was that both python-r1 eclasses would use the same approach to declaring USE dependencies between packages. This also meant that this variable must work alike on dependencies that are multi-impl and single-r1 packages. In the end, this meant a gross hack.

Without getting into details, the currently available USE dependency syntax does not permit directly depending on PT flags based on PST-based conditions. This needs to be done using the more verbose expanded syntax:

pst2_7? ( foo[pt2_7] )
pst3_7? ( foo[pt3_7] )

While this was doable back in the day, it was not possible with PYTHON_USEDEP-based approach. Hence, all single-r1 packages gained additional set of flags merely to construct dependencies conveniently.

What is the problem with that?

I suppose some of you see the problem already. Nevertheless, let’s list them explicitly.

Firstly, enabling additional implementations is inconvenient. Whenever you need to do that, you need to add both PST and PT flags.

Secondly, the PT flags are entirely redundant and meaningless for the package in question. Whenever your value of PT changes, all single-r1 packages trigger rebuilds even if their PST value stays the same.

Thirdly, the PT flags overspecify dependencies. If your PT flags specify multiple implementations (which is normally the case), all dependencies will also have to be built for those interpreters even though PST requires only one of them.

The solution

The user-visible part of the solution is that PYTHON_TARGETS are disappearing from single-r1 packages. From now on, only PYTHON_SINGLE_TARGET will be necessary. Furthermore, PT enforcement on dependencies (if necessary) will be limited to the single implementation selected by PST rather than all of PT.

The developer-oriented part is that PYTHON_USEDEP is no longer valid in single-r1 packages. Instead, PYTHON_SINGLE_USEDEP is provided for dependencies on other single-r1 packages, and PYTHON_MULTI_USEDEP placeholder is used for multi-impl packages. The former is available as a global variable, the latter only as a placeholder in python_gen_cond_dep (the name is a bit of misnomer now but I’ve decided not to introduce additional function).

All existing uses have been converted, and the eclasses will now fail if someone tries to use the old logic. The conversion of existing ebuilds is rather simple:

  1. Replace all ${PYTHON_USEDEP}s with ${PYTHON_SINGLE_USEDEP} when the dep is single-r1, or with ${PYTHON_MULTI_USEDEP} otherwise.
  2. Wrap all dependencies containing ${PYTHON_MULTI_USEDEP} in a python_gen_cond_dep. Remember that the variable must be a literal placeholder, i.e. use single quotes.

An example of the new logic follows:

RDEPEND="
  dev-libs/libfoo[python,${PYTHON_SINGLE_USEDEP}]
  $(python_gen_cond_dep '
    dev-python/foo[${PYTHON_MULTI_USEDEP}]
    dev-python/bar[${PYTHON_MULTI_USEDEP}]
  ')
"

If you get the dependency type wrong, repoman/pkgcheck will complain about bad dependency.

Since its inception in 2012, python-single-r1 has been haunting users with two sets of USE flags: PYTHON_TARGETS and PYTHON_SINGLE_TARGET. While this initially seemed a necessary part of the grand design, today I know we could have done better. Today this chymera is disappearing for real, and python-single-r1 are going to use PYTHON_SINGLE_TARGET flags only.

I would like to take this opportunity to explain why the eclass has been designed this way in the first place, and what has been done to change that.

Why PYTHON_SINGLE_TARGET?

Why did we need a second variable in the first place? After all, we could probably get away with using PYTHON_TARGETS everywhere, and adding an appropriate REQUIRED_USE constraint.

Back in the day we have established that for users’ convenience we need to default to enabling one version of Python 2 and one version of Python 3. If we enabled only one of them, the users would end up having to enable the other for a lot of packages. On the other had, if we combined both with using PT for single-r1 packages, the users would have to disable the extra implementation for a lot of them. Neither option was good.

The primary purpose of PYTHON_SINGLE_TARGET was to provide a parallel sensible setting for those packages. It was not only to make the default work out of the box but also to let users change it in one step.

Today, with the demise of Python 2 and the effort to remove Python 2 from default PT, it may seem less important to keep the distinction. Nevertheless, a number of developers and at least some users keep multiple versions of Python in PT to test their packages. Having PST is still helpful to them.

Why additional PYTHON_TARGETS then?

PST is only half of the story. What I explained above does not justify having PYTHON_TARGETS on those packages as well, and a REQUIRED_USE constraint to make them superset of enabled PST. Why did we need to have two flag sets then?

The answer is: PYTHON_USEDEP. The initial design goal was that both python-r1 eclasses would use the same approach to declaring USE dependencies between packages. This also meant that this variable must work alike on dependencies that are multi-impl and single-r1 packages. In the end, this meant a gross hack.

Without getting into details, the currently available USE dependency syntax does not permit directly depending on PT flags based on PST-based conditions. This needs to be done using the more verbose expanded syntax:

pst2_7? ( foo[pt2_7] )
pst3_7? ( foo[pt3_7] )

While this was doable back in the day, it was not possible with PYTHON_USEDEP-based approach. Hence, all single-r1 packages gained additional set of flags merely to construct dependencies conveniently.

What is the problem with that?

I suppose some of you see the problem already. Nevertheless, let’s list them explicitly.

Firstly, enabling additional implementations is inconvenient. Whenever you need to do that, you need to add both PST and PT flags.

Secondly, the PT flags are entirely redundant and meaningless for the package in question. Whenever your value of PT changes, all single-r1 packages trigger rebuilds even if their PST value stays the same.

Thirdly, the PT flags overspecify dependencies. If your PT flags specify multiple implementations (which is normally the case), all dependencies will also have to be built for those interpreters even though PST requires only one of them.

The solution

The user-visible part of the solution is that PYTHON_TARGETS are disappearing from single-r1 packages. From now on, only PYTHON_SINGLE_TARGET will be necessary. Furthermore, PT enforcement on dependencies (if necessary) will be limited to the single implementation selected by PST rather than all of PT.

The developer-oriented part is that PYTHON_USEDEP is no longer valid in single-r1 packages. Instead, PYTHON_SINGLE_USEDEP is provided for dependencies on other single-r1 packages, and PYTHON_MULTI_USEDEP placeholder is used for multi-impl packages. The former is available as a global variable, the latter only as a placeholder in python_gen_cond_dep (the name is a bit of misnomer now but I’ve decided not to introduce additional function).

All existing uses have been converted, and the eclasses will now fail if someone tries to use the old logic. The conversion of existing ebuilds is rather simple:

  1. Replace all ${PYTHON_USEDEP}s with ${PYTHON_SINGLE_USEDEP} when the dep is single-r1, or with ${PYTHON_MULTI_USEDEP} otherwise.
  2. Wrap all dependencies containing ${PYTHON_MULTI_USEDEP} in a python_gen_cond_dep. Remember that the variable must be a literal placeholder, i.e. use single quotes.

An example of the new logic follows:

RDEPEND="
  dev-libs/libfoo[python,${PYTHON_SINGLE_USEDEP}]
  $(python_gen_cond_dep '
    dev-python/foo[${PYTHON_MULTI_USEDEP}]
    dev-python/bar[${PYTHON_MULTI_USEDEP}]
  ')
"

If you get the dependency type wrong, repoman/pkgcheck will complain about bad dependency.

January 03 2020

FOSDEM 2020

Gentoo News (GentooNews) January 03, 2020, 6:00

It’s FOSDEM time again! Join us at Université libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year’s FOSDEM 2020 will be held on February 1st and 2nd.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building K where we will also celebrate 20 years compiling! Visit this year’s wiki page to see who’s coming.

FOSDEM logo

It’s FOSDEM time again! Join us at Université libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year’s FOSDEM 2020 will be held on February 1st and 2nd.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building K where we will also celebrate 20 years compiling! Visit this year’s wiki page to see who’s coming.

December 28 2019

Scylla Summit 2019

Alexys Jacob (ultrabug) December 28, 2019, 19:04

I’ve had the pleasure to attend again and present at the Scylla Summit in San Francisco and the honor to be awarded the Most innovative use case of Scylla.

It was a great event, full of friendly people and passionate conversations. Peter did a great full write-up of it already so I wanted to share some of my notes instead…

This a curated set of topics that I happened to question or discuss in depth so this post is not meant to be taken as a full coverage of the conference.

Scylla Manager version 2

The upcoming version of scylla-manager is dropping its dependency on SSH setup which will be replaced by an agent, most likely shipped as a separate package.

On the features side, I was a bit puzzled by the fact that ScyllaDB is advertising that its manager will provide a repair scheduling window so that you can control when it’s running or not.

Why did it struck me you ask?

Because MongoDB does the same thing within its balancer process and I always thought of this as a patch to a feature that the database should be able to cope with by itself.

And that database-do-it-better-than-you motto is exactly one of the promises of Scylla, the boring database, so smart at handling workload impacts on performance that you shouldn’t have to start playing tricks to mitigate them… I don’t want this time window feature on scylla-manager to be a trojan horse on the demise of that promise!

Kubernetes

They almost got late on this but are working hard to play well with the new toy of every tech around the world. Helm charts are also being worked on!

The community developed scylla operator by Yannis is now being worked on and backed by ScyllaDB. It can deploy, scale up and down a cluster.

Few things to note:

  • it’s using a configmap to store the scylla config
  • no TLS support yet
  • no RBAC support yet
  • kubernetes networking is lighter on the network performance hit that was seen on Docker
  • use placement strategies to dedicate kubernetes nodes to scylla!
Change Data Capture

Oh boy this one was awaited… but it’s now coming soon!

I inquired about it’s performance impact since every operation will be written to a table. Clearly my questioning was a bit alpha since CDC is still being worked on.

I had the chance to discuss ideas with Kamil, Tzach and Dor: one of the thing that one of my colleague Julien asked for was the ability for the CDC to generate an event when a tombstone is written so we could actually know when a specific data expired!

I want to stress a few other things too:

  • default TTL on CDC table is 24H
  • expect I/O impact (logical)
  • TTL tombstones can have a hidden disk space cost and nobody was able to tell me if the CDC table was going to be configured with a lower gc_grace_period than the default 10 days so that’s something we need to keep in mind and check for
  • there was no plan to add user information that would allow us to know who actually did the operation, so that’s something I asked for because it could be used as a cheap and open source way to get auditing!
LightWeight Transactions

Another so long awaited feature is also coming from the amazing work and knowledge of Konstantin. We had a great conversation about the differences between the currently worked on Paxos based LWT implementation and the maybe later Raft one.

So yes, the first LWT implementation will be using Paxos as a consensus algorithm. This will make the LWT feature very consistent while having it slower that what could be achieved using Raft. That’s why ScyllaDB have plans on another implementation that could be faster with less data consistency guarantees.

User Defined Functions / Aggregations

This one is bringing the Lua language inside Scylla!

To be precise, it will be a Lua JIT as its footprint is low and Lua can be cooperative enough but the ScyllaDB people made sure to monitor its violations (when it should yield but does not) and act strongly upon them.

I got into implementation details with Avi, this is what I noted:

  • lua function return type is not checked at creation but at execution, so expect runtime errors if your lua code is bad
  • since lua is lightweight, there’s no need to assign a core to lua execution
  • I found UDA examples, like top-k rows, to be very similar to the Map/Reduce logic
  • UDF will allow simpler token range full table scans thanks to syntax sugar
  • there will be memory limits applied to result sets from UDA, and they will be tunable
Text search

Dejan is the text search guy at ScyllaDB and the one who kindly implemented the LIKE feature we asked for and that will be released in the upcoming 3.2 version.

We discussed ideas and projected use cases to make sure that what’s going to be worked on will be used!

Redis API

I’ve always been frustrated about Redis because while I love the technology I never trusted its clustering and scaling capabilities.

What if you could scale your Redis like Scylla without giving up on performance? That’s what the implementation of the Redis API backed by Scylla will get us!

I’m desperately looking forward to see this happen!

ultrabug (ultrabug ) December 28, 2019, 19:04

I’ve had the pleasure to attend again and present at the Scylla Summit in San Francisco and the honor to be awarded the Most innovative use case of Scylla.

It was a great event, full of friendly people and passionate conversations. Peter did a great full write-up of it already so I wanted to share some of my notes instead…

This a curated set of topics that I happened to question or discuss in depth so this post is not meant to be taken as a full coverage of the conference.

Scylla Manager version 2

The upcoming version of scylla-manager is dropping its dependency on SSH setup which will be replaced by an agent, most likely shipped as a separate package.

On the features side, I was a bit puzzled by the fact that ScyllaDB is advertising that its manager will provide a repair scheduling window so that you can control when it’s running or not.

Why did it struck me you ask?

Because MongoDB does the same thing within its balancer process and I always thought of this as a patch to a feature that the database should be able to cope with by itself.

And that database-do-it-better-than-you motto is exactly one of the promises of Scylla, the boring database, so smart at handling workload impacts on performance that you shouldn’t have to start playing tricks to mitigate them… I don’t want this time window feature on scylla-manager to be a trojan horse on the demise of that promise!

Kubernetes

They almost got late on this but are working hard to play well with the new toy of every tech around the world. Helm charts are also being worked on!

The community developed scylla operator by Yannis is now being worked on and backed by ScyllaDB. It can deploy, scale up and down a cluster.

Few things to note:

  • it’s using a configmap to store the scylla config
  • no TLS support yet
  • no RBAC support yet
  • kubernetes networking is lighter on the network performance hit that was seen on Docker
  • use placement strategies to dedicate kubernetes nodes to scylla!

Change Data Capture

Oh boy this one was awaited… but it’s now coming soon!

I inquired about it’s performance impact since every operation will be written to a table. Clearly my questioning was a bit alpha since CDC is still being worked on.

I had the chance to discuss ideas with Kamil, Tzach and Dor: one of the thing that one of my colleague Julien asked for was the ability for the CDC to generate an event when a tombstone is written so we could actually know when a specific data expired!

I want to stress a few other things too:

  • default TTL on CDC table is 24H
  • expect I/O impact (logical)
  • TTL tombstones can have a hidden disk space cost and nobody was able to tell me if the CDC table was going to be configured with a lower gc_grace_period than the default 10 days so that’s something we need to keep in mind and check for
  • there was no plan to add user information that would allow us to know who actually did the operation, so that’s something I asked for because it could be used as a cheap and open source way to get auditing!

LightWeight Transactions

Another so long awaited feature is also coming from the amazing work and knowledge of Konstantin. We had a great conversation about the differences between the currently worked on Paxos based LWT implementation and the maybe later Raft one.

So yes, the first LWT implementation will be using Paxos as a consensus algorithm. This will make the LWT feature very consistent while having it slower that what could be achieved using Raft. That’s why ScyllaDB have plans on another implementation that could be faster with less data consistency guarantees.

User Defined Functions / Aggregations

This one is bringing the Lua language inside Scylla!

To be precise, it will be a Lua JIT as its footprint is low and Lua can be cooperative enough but the ScyllaDB people made sure to monitor its violations (when it should yield but does not) and act strongly upon them.

I got into implementation details with Avi, this is what I noted:

  • lua function return type is not checked at creation but at execution, so expect runtime errors if your lua code is bad
  • since lua is lightweight, there’s no need to assign a core to lua execution
  • I found UDA examples, like top-k rows, to be very similar to the Map/Reduce logic
  • UDF will allow simpler token range full table scans thanks to syntax sugar
  • there will be memory limits applied to result sets from UDA, and they will be tunable

Text search

Dejan is the text search guy at ScyllaDB and the one who kindly implemented the LIKE feature we asked for and that will be released in the upcoming 3.2 version.

We discussed ideas and projected use cases to make sure that what’s going to be worked on will be used!

Redis API

I’ve always been frustrated about Redis because while I love the technology I never trusted its clustering and scaling capabilities.

What if you could scale your Redis like Scylla without giving up on performance? That’s what the implementation of the Redis API backed by Scylla will get us!

I’m desperately looking forward to see this happen!

December 24 2019

Handling PEP 517 (pyproject.toml) packages in Gentoo

Michał Górny (mgorny) December 24, 2019, 22:59

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that.

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it’s limited to building wheel packages and source distribution tarballs.

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we’ve decided to try an alternate route.

Instead of trying to tame the new build systems, or work around their deficiencies (i.e. by making them build wheel packages, then unpacking and repackaging them), we’ve explored the possibility of converting the pyproject.toml files into setup.py scripts. Since the new formats are declarative, this should not be that hard.

We’ve found poetry-setup project which seemed to have a similar goal. However, it was already discontinued at the time in favor of dephell. The latter project looked pretty powerful but the name was pretty ominous. We did not need most of the functions, and it was hell to package.

Finally, I’ve managed to dedicate some time into building an in-house solution instead. pyproject2setuppy is a small-ish (<100 SLOC) pyproject.toml-to-setuptools adapter which allows us to run flit- or poetry-based projects as if they used regular distutils. While it’s quite limited, it’s good enough to build and install the packages that we needed to deal with so far.

The design is quite simple — it reads pyproject.toml and calls setuptools’ setup() function with the metadata read. As such, the package can even be used to provide a backwards-compatible setup.py script in other packages. In fact, this is how its own setup.py works — it carries flit-compatible pyproject.toml and uses itself to install itself via setuptools.

dev-python/pyproject2setuppy is already packaged in Gentoo. I’ve sent eclass patches to easily integrate it into distutils-r1. Once they are merged, installing pyproject.toml packages should be as simple as adding the following declaration into ebuilds:

DISTUTILS_USE_SETUPTOOLS=pyproject.toml

This should make things easier both for us (as it saves us from having to hurriedly add new build systems and their NIH dependencies) and for users who will not have to suffer from more circular dependencies in the Python world. It may also help some upstream projects to maintain backwards compatibility while migrating to new build systems.

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that.

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it’s limited to building wheel packages and source distribution tarballs.

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we’ve decided to try an alternate route.

Instead of trying to tame the new build systems, or work around their deficiencies (i.e. by making them build wheel packages, then unpacking and repackaging them), we’ve explored the possibility of converting the pyproject.toml files into setup.py scripts. Since the new formats are declarative, this should not be that hard.

We’ve found poetry-setup project which seemed to have a similar goal. However, it was already discontinued at the time in favor of dephell. The latter project looked pretty powerful but the name was pretty ominous. We did not need most of the functions, and it was hell to package.

Finally, I’ve managed to dedicate some time into building an in-house solution instead. pyproject2setuppy is a small-ish (<100 SLOC) pyproject.toml-to-setuptools adapter which allows us to run flit- or poetry-based projects as if they used regular distutils. While it’s quite limited, it’s good enough to build and install the packages that we needed to deal with so far.

The design is quite simple — it reads pyproject.toml and calls setuptools’ setup() function with the metadata read. As such, the package can even be used to provide a backwards-compatible setup.py script in other packages. In fact, this is how its own setup.py works — it carries flit-compatible pyproject.toml and uses itself to install itself via setuptools.

dev-python/pyproject2setuppy is already packaged in Gentoo. I’ve sent eclass patches to easily integrate it into distutils-r1. Once they are merged, installing pyproject.toml packages should be as simple as adding the following declaration into ebuilds:

DISTUTILS_USE_SETUPTOOLS=pyproject.toml

This should make things easier both for us (as it saves us from having to hurriedly add new build systems and their NIH dependencies) and for users who will not have to suffer from more circular dependencies in the Python world. It may also help some upstream projects to maintain backwards compatibility while migrating to new build systems.

December 19 2019

A distribution kernel for Gentoo

Michał Górny (mgorny) December 19, 2019, 12:32

The traditional Gentoo way of getting a kernel is to install the sources, and then configure and build one yourself. For those who didn’t want to go through the tedious process of configuring it manually, an alternative route of using genkernel was provided. However, neither of those variants was able to really provide the equivalent of kernels provided by binary distributions.

I have manually configured the kernels for my private systems long time ago. Today, I wouldn’t really have bothered. In fact, I realized that for some time I’m really hesitant to even upgrade them because of the effort needed to update configuration. The worst part is, whenever a new kernel does not boot, I have to ask myself: is it a real bug, or is it my fault for configuring it wrong?

I’m not alone in this. Recently Михаил Коляда (zlogene) has talked to me about providing binary kernels for Gentoo. While I have not strictly implemented what he had in mind, he inspired me to start working on a distribution kernel. The goal was to create a kernel package that users can install to get a working kernel with minimal effort, and that would be upgraded automatically as part of regular @world upgrades.

Pros and cons of your own kernel

If I am to justify switching from the old tradition of custom kernels to a universal kernel package, I should start by discussing the reasons why you may want to configure a custom kernel in the first place.

In my opinion, the most important feature of a custom kernel is that you can fine-tune it to your hardware. You just have to build the drivers you need (or may need), and the features you care about. The modules for my last custom kernel have occupied 44 MiB. The modules for the distribution kernel occupy 294 MiB. Such a difference in size also comes with a proportional increase of build time. This can be an important argument for people with low-end hardware. On the other hand, the distribution kernel permits building reusable binary packages that can save more computing power.

The traditional Gentoo argument is performance. However, these days I would be very careful arguing about that. I suppose you are able to reap benefits if you know how to configure your kernel towards a specific workload. But then — a misconfiguration can have the exact opposite effect. We must not forget that binary distributions are important players in the field — and the kernel must also be able to achieve good performance when not using a dedicated configuration.

At some point I have worked on achieving a very fast startup. For this reason I’ve switched to using LILO as the bootloader, and a kernel suitable for booting my system without an initramfs. A universal kernel naturally needs an initramfs, and is slower to boot.

The main counterargument is the effort. As mentioned earlier, I’ve personally grown tired of having to manually deal with my kernel. Do the potential gains mentioned outweigh the loss of human time on configuring and maintaining a custom kernel?

Creating a truly universal kernel

A distribution kernel makes sense only if it works on a wide range of systems. Furthermore, I didn’t forget the original idea of binary kernel packages. I didn’t want to write an ebuild that can install a working kernel anywhere. I wanted to create an ebuild that can be used to build a binary package that’s going to work on a wide range of setups — including not only different hardware but also bootloaders and /boot layout. A package that would work fine both for my ‘traditional’ LILO setup and UEFI systemd-boot setup.

The first part of a distribution kernel is the right configuration. I wanted to use a well-tested configuration known to build kernels used on many systems, while at the same time minimizing the maintenance effort on our end. Reusing the configuration from a binary distro was the obvious solution. I went for using the config from Arch Linux’s kernel package with minimal changes (e.g. changing the default hostname to Gentoo).

The second part is an initramfs. Since we need to support a wide variety of setups, we can’t get away without it. To follow the configuration used, Dracut was the natural choice.

The third and hardest part is installing it all. Since I’ve already set a goal of reusing the same binary package on different filesystem layouts, the actual installation needed to be moved to postinst phase. Our distribution kernel package installs the kernel into an interim location which is entirely setup-independent, rendering the binary packages setup-agnostic as well. The initramfs is created and installed into the final location along with the kernel in pkg_postinst.

Support for different install layouts is provided by reusing the installkernel tool, originally installed by debianutils. As part of the effort, it was extended with initramfs support and moved into a separate sys-kernel/installkernel-gentoo package. Furthermore, an alternative sys-kernel/installkernel-systemd-boot package was created to provide an out-of-the-box support for systemd-boot layout. If neither of those two work for you, you can easily create your own /usr/local/bin/installkernel that follows your own layout.

Summary

The experimental versions of the distribution kernel are packaged as sys-kernel/vanilla-kernel (in distinction from sys-kernel/vanilla-sources that install the sources). Besides providing the default zero-effort setup, the package supports using your own configuration via savedconfig (but no easy way to update it at the moment). It also provides a forced flag that can be used by expert users to disable the initramfs.

The primary goal at the moment is to test the package and find bugs that could prevent our users from using it. In the future, we’re planning to extend it to other architectures, kernel variants (Gentoo patch set in particular) and LTS versions. We’re also considering providing prebuilt binary packages — however, this will probably be a part of a bigger effort into providing an official Gentoo binhost.

The traditional Gentoo way of getting a kernel is to install the sources, and then configure and build one yourself. For those who didn’t want to go through the tedious process of configuring it manually, an alternative route of using genkernel was provided. However, neither of those variants was able to really provide the equivalent of kernels provided by binary distributions.

I have manually configured the kernels for my private systems long time ago. Today, I wouldn’t really have bothered. In fact, I realized that for some time I’m really hesitant to even upgrade them because of the effort needed to update configuration. The worst part is, whenever a new kernel does not boot, I have to ask myself: is it a real bug, or is it my fault for configuring it wrong?

I’m not alone in this. Recently Михаил Коляда (zlogene) has talked to me about providing binary kernels for Gentoo. While I have not strictly implemented what he had in mind, he inspired me to start working on a distribution kernel. The goal was to create a kernel package that users can install to get a working kernel with minimal effort, and that would be upgraded automatically as part of regular @world upgrades.

Pros and cons of your own kernel

If I am to justify switching from the old tradition of custom kernels to a universal kernel package, I should start by discussing the reasons why you may want to configure a custom kernel in the first place.

In my opinion, the most important feature of a custom kernel is that you can fine-tune it to your hardware. You just have to build the drivers you need (or may need), and the features you care about. The modules for my last custom kernel have occupied 44 MiB. The modules for the distribution kernel occupy 294 MiB. Such a difference in size also comes with a proportional increase of build time. This can be an important argument for people with low-end hardware. On the other hand, the distribution kernel permits building reusable binary packages that can save more computing power.

The traditional Gentoo argument is performance. However, these days I would be very careful arguing about that. I suppose you are able to reap benefits if you know how to configure your kernel towards a specific workload. But then — a misconfiguration can have the exact opposite effect. We must not forget that binary distributions are important players in the field — and the kernel must also be able to achieve good performance when not using a dedicated configuration.

At some point I have worked on achieving a very fast startup. For this reason I’ve switched to using LILO as the bootloader, and a kernel suitable for booting my system without an initramfs. A universal kernel naturally needs an initramfs, and is slower to boot.

The main counterargument is the effort. As mentioned earlier, I’ve personally grown tired of having to manually deal with my kernel. Do the potential gains mentioned outweigh the loss of human time on configuring and maintaining a custom kernel?

Creating a truly universal kernel

A distribution kernel makes sense only if it works on a wide range of systems. Furthermore, I didn’t forget the original idea of binary kernel packages. I didn’t want to write an ebuild that can install a working kernel anywhere. I wanted to create an ebuild that can be used to build a binary package that’s going to work on a wide range of setups — including not only different hardware but also bootloaders and /boot layout. A package that would work fine both for my ‘traditional’ LILO setup and UEFI systemd-boot setup.

The first part of a distribution kernel is the right configuration. I wanted to use a well-tested configuration known to build kernels used on many systems, while at the same time minimizing the maintenance effort on our end. Reusing the configuration from a binary distro was the obvious solution. I went for using the config from Arch Linux’s kernel package with minimal changes (e.g. changing the default hostname to Gentoo).

The second part is an initramfs. Since we need to support a wide variety of setups, we can’t get away without it. To follow the configuration used, Dracut was the natural choice.

The third and hardest part is installing it all. Since I’ve already set a goal of reusing the same binary package on different filesystem layouts, the actual installation needed to be moved to postinst phase. Our distribution kernel package installs the kernel into an interim location which is entirely setup-independent, rendering the binary packages setup-agnostic as well. The initramfs is created and installed into the final location along with the kernel in pkg_postinst.

Support for different install layouts is provided by reusing the installkernel tool, originally installed by debianutils. As part of the effort, it was extended with initramfs support and moved into a separate sys-kernel/installkernel-gentoo package. Furthermore, an alternative sys-kernel/installkernel-systemd-boot package was created to provide an out-of-the-box support for systemd-boot layout. If neither of those two work for you, you can easily create your own /usr/local/bin/installkernel that follows your own layout.

Summary

The experimental versions of the distribution kernel are packaged as sys-kernel/vanilla-kernel (in distinction from sys-kernel/vanilla-sources that install the sources). Besides providing the default zero-effort setup, the package supports using your own configuration via savedconfig (but no easy way to update it at the moment). It also provides a forced flag that can be used by expert users to disable the initramfs.

The primary goal at the moment is to test the package and find bugs that could prevent our users from using it. In the future, we’re planning to extend it to other architectures, kernel variants (Gentoo patch set in particular) and LTS versions. We’re also considering providing prebuilt binary packages — however, this will probably be a part of a bigger effort into providing an official Gentoo binhost.

VIEW

SCOPE

FILTER
  from
  to