Gentoo Logo
Gentoo Logo Side
Gentoo Spaceship

Contributors:
. Aaron W. Swenson
. Agostino Sarubbo
. Alexey Shvetsov
. Alexis Ballier
. Alexys Jacob
. Alice Ferrazzi
. Alice Ferrazzi
. Andreas K. Hüttel
. Anthony Basile
. Arun Raghavan
. Bernard Cafarelli
. Brian Harring
. Christian Ruppert
. Chí-Thanh Christopher Nguyễn
. Denis Dupeyron
. Detlev Casanova
. Diego E. Pettenò
. Domen Kožar
. Doug Goldstein
. Eray Aslan
. Erik Mackdanz
. Fabio Erculiani
. Gentoo Haskell Herd
. Gentoo Miniconf 2016
. Gentoo Monthly Newsletter
. Gentoo News
. Gilles Dartiguelongue
. Greg KH
. Göktürk Yüksek
. Hanno Böck
. Hans de Graaff
. Ian Whyman
. Jan Kundrát
. Jason A. Donenfeld
. Jeffrey Gardner
. Joachim Bartosik
. Johannes Huber
. Jonathan Callen
. Jorge Manuel B. S. Vicetto
. Kristian Fiskerstrand
. Liam McLoughlin
. Luca Barbato
. Marek Szuba
. Mart Raudsepp
. Matt Turner
. Matthew Thode
. Michael Palimaka
. Michal Hrusecky
. Michał Górny
. Mike Doty
. Mike Gilbert
. Mike Pagano
. Nathan Zachary
. Pacho Ramos
. Patrick Kursawe
. Patrick Lauer
. Patrick McLean
. Paweł Hajdan, Jr.
. Petteri Räty
. Piotr Jaroszyński
. Rafael G. Martins
. Remi Cardona
. Richard Freeman
. Robin Johnson
. Sean Amoss
. Sebastian Pipping
. Steev Klimaszewski
. Sven Vermeulen
. Sven Wegener
. Tom Wijsman
. Tomáš Chvátal
. Yury German
. Zack Medico

Last updated:
September 26, 2017, 12:04 UTC

Disclaimer:
Views expressed in the content published here do not necessarily represent the views of Gentoo Linux or the Gentoo Foundation.


Bugs? Comments? Suggestions? Contact us!

Powered by:
Planet Venus

Welcome to Planet Gentoo, an aggregation of Gentoo-related weblog articles written by Gentoo developers. For a broader range of topics, you might be interested in Gentoo Universe.

September 11, 2017
Sven Vermeulen a.k.a. swift (homepage, bugs)
Authenticating with U2F (September 11, 2017, 16:25 UTC)

In order to further secure access to my workstation, after the switch to Gentoo sources, I now enabled two-factor authentication through my Yubico U2F USB device. Well, at least for local access - remote access through SSH requires both userid/password as well as the correct SSH key, by chaining authentication methods in OpenSSH.

Enabling U2F on (Gentoo) Linux is fairly easy. The various guides online which talk about the pam_u2f setup are indeed correct that it is fairly simple. For completeness sake, I've documented what I know on the Gentoo Wiki, as the pam_u2f article.

The setup, basically

The setup of U2F is done in a number of steps: 1. Validate that the kernel is ready for the USB device 2. Install the PAM module and supporting tools 3. Generate the necessary data elements for each user (keys and such) 4. Configure PAM to require authentication through the U2F key

For the kernel, the configuration item needed is the raw HID device support. Now, in current kernels, two settings are available that both talk about raw HID device support: CONFIG_HIDRAW is the general raw HID device support, while CONFIG_USB_HIDDEV is the USB-specific raw HID device support.

It is very well possible that only a single one is needed, but both where active on my kernel configuration already, and Internet sources are not clear which one is needed, so let's assume for now both are.

Next, the PAM module needs to be installed. On Gentoo, this is a matter of installing the pam\_u2f package, as the necessary dependencies will be pulled in automatically:

~# emerge pam_u2f

Next, for each user, a registration has to be made. This registration is needed for the U2F components to be able to correctly authenticate the use of a U2F key for a particular user. This is done with pamu2fcfg:

~$ pamu2fcfg -u<username> > ~/.config/Yubico/u2f_keys

The U2F USB key must be plugged in when the command is executed, as a succesful keypress (on the U2F device) is needed to complete the operation.

Finally, enable the use of the pam\_u2f module in PAM. On my system, this is done through the /etc/pam.d/system-local-login PAM configuration file used by all local logon services.

auth     required     pam_u2f.so

Consider the problems you might face

When fiddling with PAM, it is important to keep in mind what could fail. During the setup, it is recommended to have an open administrative session on the system so that you can validate if the PAM configuration works, without locking yourself out of the system.

But other issues need to be considered as well.

My Yubico U2F USB key might have a high MTBF (Mean Time Between Failures) value, but once it fails, it would lock me out of my workstation (and even remote services and servers that use it). For that reason, I own a second one, safely stored, but is a valid key nonetheless for my workstation and remote systems/services. Given the low cost of a simple U2F key, it is a simple solution for this threat.

Another issue that could come up is a malfunction in the PAM module itself. For me, this is handled by having remote SSH access done without this PAM module (although other PAM modules are still involved, so a generic PAM failure itself wouldn't resolve this). Of course, worst case, the system needs to be rebooted in single user mode.

One issue that I faced was the SELinux policy. Some applications that provide logon services don't have the proper rights to handle U2F, and because PAM just works in the address space (and thus SELinux domain) of the application, the necessary privileges need to be added to these services. My initial investigation revealed the following necessary policy rules (refpolicy-style);

udev_search_pids(...)
udev_read_db(...)
dev_rw_generic_usb_dev(...)

The first two rules are needed because the operation to trigger the USB key uses the udev tables to find out where the key is located/attached, before it interacts with it. This interaction is then controlled through the first rule.

Simple yet effective

Enabling the U2F authentication on the system is very simple, and gives a higher confidence that malicious activities through regular accounts will have it somewhat more challenging to switch to a more privileged session (one control is the SELinux policy of course, but for those domains that are allowed to switch then the PAM-based authentication is another control), as even evesdropping on my password (or extracting it from memory) won't suffice to perform a successful authentication.

If you want to use a different two-factor authentication, check out the use of the Google authenticator, another nice article on the Gentoo wiki. It is also possible to use Yubico keys for remote authentication, but that uses the OTP (One Time Password) functionality which isn't active on the Yubico keys that I own.

September 06, 2017
Greg KH a.k.a. gregkh (homepage, bugs)
4.14 == This years LTS kernel (September 06, 2017, 14:41 UTC)

As the 4.13 release has now happened, the merge window for the 4.14 kernel release is now open. I mentioned this many weeks ago, but as the word doesn’t seem to have gotten very far based on various emails I’ve had recently, I figured I need to say it here as well.

So, here it is officially, 4.14 should be the next LTS kernel that I’ll be supporting with stable kernel patch backports for at least two years, unless it really is a horrid release and has major problems. If so, I reserve the right to pick a different kernel, but odds are, given just how well our development cycle has been going, that shouldn’t be a problem (although I guess I just doomed it now…)

As always, if people have questions about this, email me and I will be glad to discuss it, or talk to me in person next week at the LinuxCon^WOpenSourceSummit or Plumbers conference in Los Angeles, or at any of the other conferences I’ll be at this year (ELCE, Kernel Recipes, etc.)

September 04, 2017
Arun Raghavan a.k.a. ford_prefect (homepage, bugs)
A Late GUADEC 2017 Post (September 04, 2017, 03:20 UTC)

It’s been a little over a month since I got back from Manchester, and this post should’ve come out earlier but I’ve been swamped.

The conference was absolutely lovely, the organisation was a 110% on point (serious kudos, I know first hand how hard that is). Others on Planet GNOME have written extensively about the talks, the social events, and everything in between that made it a great experience. What I would like to write about is about why this year’s GUADEC was special to me.

GNOME turning 20 years old is obviously a large milestone, and one of the main reasons I wanted to make sure I was at Manchester this year. There were many occasions to take stock of how far we had come, where we are, and most importantly, to reaffirm who we are, and why we do what we do.

And all of this made me think of my own history with GNOME. In 2002/2003, Nat and Miguel came down to Bangalore to talk about some of the work they were doing. I know I wasn’t the only one who found their energy infectious, and at Linux Bangalore 2003, they got on stage, just sat down, and started hacking up a GtkMozEmbed-based browser. The idea itself was fun, but what I took away — and I know I wasn’t the only one — is the sheer inclusive joy they shared in creating something and sharing that with their audience.

For all of us working on GNOME in whatever way we choose to contribute, there is the immediate gratification of shaping this project, as well as the larger ideological underpinning of making everyone’s experience talking to their computers better and free-er.

But I think it is also important to remember that all our efforts to make our community an inviting and inclusive space have a deep impact across the world. So much so that complete strangers from around the world are able to feel a sense of belonging to something much larger than themselves.

I am excited about everything we will achieve in the next 20 years.

(thanks go out to the GNOME Foundation for helping me attend GUADEC this year)

Sponsored by GNOME!

August 26, 2017
Sebastian Pipping a.k.a. sping (homepage, bugs)
GIMP 2.9.6 now in Gentoo (August 26, 2017, 19:53 UTC)

Here’s what upstream has to say about the new release 2.9.6. Enjoy 🙂

August 23, 2017
Sven Vermeulen a.k.a. swift (homepage, bugs)
Using nVidia with SELinux (August 23, 2017, 17:04 UTC)

Yesterday I've switched to the gentoo-sources kernel package on Gentoo Linux. And with that, I also attempted (succesfully) to use the propriatary nvidia drivers so that I can enjoy both a smoother 3D experience while playing minecraft, as well as use the CUDA support so I don't need to use cloud-based services for small exercises.

The move to nvidia was quite simple, as the nvidia-drivers wiki article on the Gentoo wiki was quite easy to follow.

August 22, 2017
Sven Vermeulen a.k.a. swift (homepage, bugs)
Switch to Gentoo sources (August 22, 2017, 17:04 UTC)

You've might already read it on the Gentoo news site, the Hardened Linux kernel sources are removed from the tree due to the grsecurity change where the grsecurity Linux kernel patches are no longer provided for free. The decision was made due to supportability and maintainability reasons.

That doesn't mean that users who want to stick with the grsecurity related hardening features are left alone. Agostino Sarubbo has started providing sys-kernel/grsecurity-sources for the users who want to stick with it, as it is based on minipli's unofficial patchset. I seriously hope that the patchset will continue to be maintained and, who knows, even evolve further.

Personally though, I'm switching to the Gentoo sources, and stick with SELinux as one of the protection measures. And with that, I might even start using my NVidia graphics card a bit more, as that one hasn't been touched in several years (I have an Optimus-capable setup with both an Intel integrated graphics card and an NVidia one, but all attempts to use nouveau for the one game I like to play - minecraft - didn't work out that well).

Alexys Jacob a.k.a. ultrabug (homepage, bugs)
py3status v3.6 (August 22, 2017, 06:00 UTC)

After four months of cool contributions and hard work on normalization and modules’ clean up, I’m glad to announce the release of py3status v3.6!

Milestone 3.6 was mainly focused about existing modules, from their documentation to their usage of the py3 helper to streamline their code base.

Other improvements were made about error reporting while some sneaky bugs got fixed along the way.

Highlights

Not an extensive list, check the changelog.

  • LOTS of modules streamlining (mainly the hard work of @lasers)
  • error reporting improvements
  • py3-cmd performance improvements

New modules

  • i3blocks support (yes, py3status can now wrap i3blocks thanks to @tobes)
  • cmus module: to control your cmus music player, by @lasers
  • coin_market module: to display custom cryptocurrency data, by @lasers
  • moc module: to control your moc music player, by @lasers

Milestone 3.7

This milestone will give a serious kick into py3status performance. We’ll do lots of profiling and drastic work to reduce py3status CPU and memory footprints!

For now we’ve been relying a lot on threads, which is simple to operate but not that CPU/memory friendly. Since i3wm users rightly care for their efficiency we think it’s about time we address this kind of points in py3status.

Stay tuned, we have some nice ideas in stock 🙂

Thanks contributors!

This release is their work, thanks a lot guys!

  • aethelz
  • alexoneill
  • armandg
  • Cypher1
  • docwalter
  • enguerrand
  • fmorgner
  • guiniol
  • lasers
  • markrileybot
  • maximbaz
  • tablet-mode
  • paradoxisme
  • ritze
  • rixx
  • tobes
  • valdur55
  • vvoland
  • yabbes

August 21, 2017
sys-kernel/grsecurity-sources available! (August 21, 2017, 15:11 UTC)

Is known that the grsecurity project since few weeks made available the grsecurity patches only for their customers. In the meantime some people made their fork of the latest publicly available patches.

At Gentoo, for some reasons (which I respect) explained by the news item and on the mailing lists, the maintainer decided to drop the hardened-sources package at the end of September 2017

Then, I decided to make my own ebuild that uses the Genpatches plus the Unofficial forward ports of the last publicly available grsecurity patch.

Before you wondering about the code of the ebuild, let me explain the logic used:

1) The ebuild was done in this way because the version bump should result in a copy-paste on the ebuild side.
2) I don’t use the GENPATCHES variable from the kernel eclass because of the previously explained point 1.
3) I generate the tarball via a bash script which takes the genpatches, take the unofficial-grsecurity-patches and deletes the unwanted patches from the genpatches tarball (i.e. in hardened-sources we had UNIPATCH_EXCLUDE=”1500_XATTR_USER_PREFIX.patch 2900_dev-root-proc-mount-fix.patch”).
4) I don’t use the UNIPATCH_EXCLUDE variable because because of the previously explained point 3.

Don’t expect a version bump on each minor release unless there are critical bugs and/or dangerous security bugs. So please not file version bump requests on bugzilla.

If you have any issue regarding grsecurity itself, please file a bug on the github issue tracker and if you will mention the issue elsewhere, please specify that the issue is with the unofficial grsecurity port. This will avoid to “damage” the grsecurity image/credibility.

The ebuild is available into my overlay
If you have trouble on how to install that ebuild, please follow the layman article on our wiki, basically you need:

root ~ $ layman -S && layman -a ago

USE IT AT YOUR OWN RISK 😉

August 19, 2017
Hardened Linux kernel sources removal (August 19, 2017, 00:00 UTC)

As you may know the core of sys-kernel/hardened-sources has been the grsecurity patches. Recently the grsecurity developers have decided to limit access to these patches. As a result, the Gentoo Hardened team is unable to ensure a regular patching schedule and therefore the security of the users of these kernel sources. Thus, we will be masking hardened-sources on the 27th of August and will proceed to remove them from the main ebuild repository by the end of September. We recommend to use sys-kernel/gentoo-sources instead. Userspace hardening and support for SELinux will of course remain in the Gentoo ebuild repository. Please see the full news item for additional information and links.

August 18, 2017

FroSCon logo

Upcoming weekend, 19-20th August 2017, there will be a Gentoo booth again at the FrOSCon “Free and Open Source Conference” 12, in St. Augustin near Bonn! Visitors can see Gentoo live in action, get Gentoo swag, and prepare, configure, and compile their own Gentoo buttons. See you there!

August 08, 2017
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
ScyllaDB meets Gentoo Linux (August 08, 2017, 14:19 UTC)

I am happy to announce that my work on packaging ScyllaDB for Gentoo Linux is complete!

Happy or curious users are very welcome to share their thoughts and ping me to get it into portage (which will very likely happen).

Why Scylla?

Ever heard of the Cassandra NoSQL database and Java GC/Heap space problems?… if you do, you already get it 😉

I will not go into the details as their website does this way better than me but I got interested into Scylla because it fits the Gentoo Linux philosophy very well. If you remember my writing about packaging Rethinkdb for Gentoo Linux, I think that we have a great match with Scylla as well!

  • it is written in C++ so it plays very well with emerge
  • the code quality is so great that building it does not require heavy patching on the ebuild (feels good to be a packager)
  • the code relies on system libs instead of bundling them in the sources (hurrah!)
  • performance tuning is handled by smart scripting and automation, allowing the relationship between the project and the hardware is strong

I believe that these are good enough points to go further and that such a project can benefit from a source based distribution like Gentoo Linux. Of course compiling on multiple systems is a challenge for such a database but one does not improve by staying in their comfort zone.

Upstream & contributions

Packaging is a great excuse to get to know the source code of a project but more importantly the people behind it.

So here I got to my first contributions to Scylla to get Gentoo Linux as a detected and supported Linux distribution in the different scripts and tools used to automatically setup the machine it will run upon (fear not, I contributed bash & python, not C++)…

Even if I expected to contribute using Github PRs and got to change my habits to a git-patch+mailing list combo, I got warmly welcomed and received positive and genuine interest in the contributions. They got merged quickly and thanks to them you can install and experience Scylla in Gentoo Linux without heavy patching on our side.

Special shout out to Pekka, Avi and Vlad for their welcoming and insightful code reviews!

I’ve some open contributions about pushing further on the python code QA side to get the tools to a higher level of coding standards. Seeing how upstream is serious about this I have faith that it will get merged and a good base for other contributions.

Last note about reaching them is that I am a bit sad that they’re not using IRC freenode to communicate (I instinctively joined #scylla and found myself alone) but they’re on Slack (those “modern folks”) and pretty responsive to the mailing lists 😉

Java & Scylla

Even if scylla is a rewrite of Cassandra in C++, the project still relies on some external tools used by the Cassandra community which are written in Java.

When you install the scylla package on Gentoo, you will see that those two packages are Java based dependencies:

  • app-admin/scylla-tools
  • app-admin/scylla-jmx

It pained me a lot to package those (thanks to help of @monsieurp) but they are building and working as expected so this gets the packaging of the whole Scylla project pretty solid.

emerge dev-db/scylla

The scylla packages are located in the ultrabug overlay for now until I test them even more and ultimately put them in production. Then they’ll surely reach the portage tree with the approval of the Gentoo java team for the app-admin/ packages listed above.

I provide a live ebuild (scylla-9999 with no keywords) and ebuilds for the latest major version (2.0_rc1 at time of writing).

It’s as simple as:

$ sudo layman -a ultrabug
$ sudo emerge -a dev-db/scylla
$ sudo emerge --config dev-db/scylla

Try it out and tell me what you think, I hope you’ll start considering and using this awesome database!

July 27, 2017
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Last evening, I ran some updates on one of my servers. One of the updates was from MariaDB 10.1 to 10.2 (some minor release as well). After compiling, I went to restart, but it failed with:

# /etc/init.d/mysql start
* Checking mysqld configuration for mysql ...
[ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
[ERROR] Aborting

* mysql config check failed [ !! ]
* ERROR: mysql failed to start

I’m not sure why this just hit me now, but it looks like it is a function within the init script that’s causing it to look for files in the nonexistent directory of /usr/share/mysql/ instead of the appropriate /usr/share/mariadb/. The fast fix here (so that I could get everything back up and running as quickly as possible) was to simply symlink the directory:

cd /usr/share
ln -s mariadb/ mysql

Thereafter, MariaDB came up without any problem:

# /etc/init.d/mysql start
* Caching service dependencies ... [ ok ]
* Checking mysqld configuration for mysql ... [ ok ]
* Starting mysql ... [ ok ]
# /etc/init.d/mysql status
* status: started

I hope that information helps if you’re in a pinch and run into the same error message.

Cheers,
Zach

UPDATE: It seems as if the default locations for MySQL/MariaDB configurations have changed (in Gentoo). Please see this comment for more information about a supportable fix for this problem moving forward. Thanks to Brian Evans for the information. 🙂

July 23, 2017
Michał Górny a.k.a. mgorny (homepage, bugs)
Optimizing ccache using per-package caches (July 23, 2017, 18:03 UTC)

ccache can be of great assistance to Gentoo developers and users who frequently end up rebuilding similar versions of packages. By providing a caching compiler frontend, it can speed up builds by removing the need to build files that have not changed again. However, it uses a single common cache directory by default which can be suboptimal even if you are explicitly enabling ccache only for a subset of packages needing that.

The likeliness of cross-package ccache hits is pretty low — majority of the hits occurs within a single package. If you use a single cache directory for all affected packages, it grows pretty quick. Besides a possible performance hit from having a lot of files in every directory, this means that packages built later can shift earlier packages out of the cache, resulting in meaninglessly lost cache hits. A simple way to avoid both of the problems is to use separate ccache directories.

In my solution, a separate subdirectory of /var/cache/ccache is used for every package, named after the category, package name and slot. While the last one is not strictly necessary, it can be useful for slotted packages such as LLVM where I do not want frequently changing live package sources to shift the release versions out of the cache.

To use it, put a code similar to the following in your /etc/portage/bashrc:

if [[ ${FEATURES} == *ccache* && ${EBUILD_PHASE_FUNC} == src_* ]]; then
	if [[ ${CCACHE_DIR} == /var/cache/ccache ]]; then
		export CCACHE_DIR=/var/cache/ccache/${CATEGORY}/${PN}:${SLOT}
		mkdir -p "${CCACHE_DIR}" || die
	fi
fi

The first condition makes sure the code is only run when ccache is enabled, and only for src_* phases where we can rely on userpriv being used consistently. The second one makes sure the code only applies to a specific (my initial) value of CCACHE_DIR and therefore avoids both nesting the cache indefinitely when Portage calls subsequent phase functions, and applying the replacement if user overrides CCACHE_DIR.

You need to either adjust the value used here to the directory used on your system, or change it in your /etc/portage/make.conf:

CCACHE_DIR="/var/cache/ccache"

Once this is done, Portage should start creating separate cache directories for every package where you enable ccache. This should improve the cache hit ratio, especially if you are using ccache for large packages (why else would you need it?). However, note that you will no longer have a single cache size limit — every package will have its own limit. Therefore, you may want to reduce the limits per-package, or manually look after the cache periodically.

July 16, 2017
Michał Górny a.k.a. mgorny (homepage, bugs)
GLEP 73 check results explained (July 16, 2017, 08:40 UTC)

The pkgcheck instance run for the Repo mirror&CI project has finished gaining a full support for GLEP 73 REQUIRED_USE validation and verification today. As a result, it can report 5 new issues defined by that GLEP. In this article, I’d like to shortly summarize them and explain how to interpret and solve the reports.

Technical note: the GLEP number has not been formally assigned yet. However, since there is no other GLEP request open at the moment, I have taken the liberty of using the next free number in the implementation.

GLEP73Syntax: syntax violates GLEP 73

GLEP 73 specifies a few syntax restrictions as compared to the pretty much free-form syntax allowed by the PMS. The restrictions could be shortly summarized as:

  • ||, ^^ and ?? can not not be empty,
  • ||, ^^ and ?? can not not be nested,
  • USE-conditional groups can not be used inside ||, ^^ and ??,
  • All-of groups (expressed using parentheses without a prefix) are banned completely.

The full rationale for the restrictions, along with examples and proposed fixes is provided in the GLEP. For the purpose of this article, it is enough to say that in all the cases found, there was a simpler (more obvious) way of expressing the same constraint.

Violation of this syntax prevents pkgcheck from performing any of the remaining checks. But more importantly, the report indicates that the constraint is unnecessarily complex and could result in REQUIRED_USE mismatch messages that are unnecessarily confusing to the user. Taking a real example, compare:

  The following REQUIRED_USE flag constraints are unsatisfied:
    exactly-one-of ( ( !32bit 64bit ) ( 32bit !64bit ) ( 32bit 64bit ) )

and the effect of a valid replacement:

  The following REQUIRED_USE flag constraints are unsatisfied:
	any-of ( 64bit 32bit )

While we could debate about usefulness of the Portage output, I think it is clear that the second output is simpler to comprehend. And the best proof is that you actually need to think a bit before confirming that they’re equivalent.

GLEP73Immutability: REQUIRED_USE violates immutability rules

This one is rather simple: it means this constraint may tell user to enable (disable) a flag that is use.masked/forced. Taking a trivial example:

a? ( b )

GLEP73Immutability report will trigger if a profile masks the b flag. This means that if the user has a enabled, the PM would normally tell him to enable b as well. However, since b is masked, it can not be enabled using normal methods (we assume that altering use.mask is not normally expected).

The alternative is to disable a then. But what’s the point of letting user enable it if we afterwards tell him to disable it anyway? It is more friendly to disable both flags together, and this is pretty much what the check is about. So in this case, the solution is to mask a as well.

How to read it? Given the generic message of:

REQUIRED_USE violates immutability rules: [C] requires [E] while the opposite value is enforced by use.force/mask (in profiles: [P])

It indicates that in profiles P (a lot of profiles usually indicates you’re looking for base or top-level arch profile), E is forced or masked, and that you probably need to force/mask C appropriately as well.

GLEP73SelfConflicting: impossible self-conflicting condition

This one is going to be extremely rare. It indicates that somehow the REQUIRED_USE nested a condition and its negation, causing it to never evaluate to true. It is best explained using the following trivial example:

a? ( !a? ( b ) )

This constraint will never be enforced since a and !a can not be true simultaneously.

Is there a point in having such a report at all? Well, such a thing is extremely unlikely to happen. However, it would break the verification algorithms and so we need to account for it explicitly. Since we account for it anyway and it is a clear mistake, why not report it?

GLEP73Conflict: request for conflicting states

This warning indicates that there are at least two constraints that can apply simultaneously and request the opposite states for the same USE flag. Again, best explained on a generic example:

a? ( c ) b? ( !c )

In this example, any USE flag set with both a and b enabled could not satisfy the constraint. However, Portage will happily led us astray:

  The following REQUIRED_USE flag constraints are unsatisfied:
	a? ( c )

If we follow the advice and enable c, we get:

  The following REQUIRED_USE flag constraints are unsatisfied:
	b? ( !c )

The goal of this check is to avoid such a bad advices, and to require constraints to clearly indicate a suggested way forward. For example, the above case could be modified to:

a? ( !b c ) b? ( !c )

to indicate that a takes precedence over b, and that b should be disabled to avoid the impossible constraint. The opposite can be stated similarly — however, note that you need to reorder the constraints to make sure that the PM will get it right:

b? ( !a !c ) a? ( c )

How to read it? Given the generic message of:

REQUIRED_USE can request conflicting states: [Ci] requires [Ei] while [Cj] requires [Ej]

It means that if the user enables Ci and Cj simultaneously, the PM will request conflicting Ei and Ej. Depending on the intent, the solution might involve negating one of the conditions in the other constraint, or reworking the REQUIRED_USE towards another solution.

GLEP73BackAlteration: previous condition starts applying

This warning is the most specific and the least important from all the additions at the moment. It indicates that the specific constraint may cause a preceding condition to start to apply, enforcing additional requirements. Consider the following example:

b? ( c ) a? ( b )

If the user has only a enabled, the second rule will enforce b. Then the condition for the first rule will start matching, and additionally enforce c. Is this a problem? Usually not. However, for the purpose of GLEP 73 we prefer that the REQUIRED_USE can be enforced while processing left-to-right, in a single iteration. If a previous rule starts applying, we may need to do another iteration.

The solution is usually trivial: to reorder (swap) the constraints. However, in some cases developers seem to prefer copying the enforcements into the subsequent rule, e.g.:

b? ( c ) a? ( b c )

Either way works for the purposes of GLEP 73, though the latter increases complexity.

How to read it? Given the generic message of:

REQUIRED_USE causes a preceding condition to start applying: [Cj] enforces [Ej] which may cause preceding [Ci] enforcing [Ei] to evaluate to true

This indicates that if Cj is true, Ej needs to be true as well. Once it is true, a preceding condition of Ci may also become true, adding another requirement for Ei. To fix the issue, you need to either move the latter constraint before the former, or include the enforcement of Ei in the rule for Cj, rendering the application of the first rule unnecessary.

Constructs using ||, ^^ and ?? operators

GLEP 73 specifies a leftmost-preferred behavior for the ||, ^^ and ?? operators. It is expressed in a simple transformation into implications (USE-conditional groups). Long story short:

  • || and ^^ groups force the leftmost unmasked flag if none of the flags are enabled already, and
  • ?? and ^^ groups disable all but the leftmost enabled flag if more than one flag is enabled.

All the verification algorithms work on the transformed form, and so their output may list conditions resulting from it. For example, the following construct:

|| ( a b c ) static? ( !a )

will report a conflict between !b !c ⇒ a and static ⇒ !a. This indicates the fact that per the forementioned rule, || group is transformed into !b? ( !c? ( a ) ) which explains that if none of the flags are enabled, the first one is preferred, causing a conflict with the static flag.

In this particular case you could debate that the algorithm could choose b or c instead in order to avoid the problem. However, we determined that this kind of heuristic is not a goal for GLEP 73, and instead we always obide the developer’s preference expressed in the ordering. The only exception to this rule is when the leftmost flag can not match due to a mask, in which case the first unmasked flag is used.

For completeness, I should add that ?? and ^^ blocks create implications in the form of: a ⇒ !b !c…, b ⇒ !c… and so on.

At some point I might work on making the reports include the original form to avoid ambiguity.

The future

The most important goal for GLEP 73 is to make it possible for users to install packages out-of-the-box without having to fight through mazes of REQUIRED_USE, and for developers to use REQUIRED_USE not only sparingly but whenever possible to improve the visibility of resulting package configuration. However, there is still a lot of testing, some fixing and many bikesheds before that could happen.

Nevertheless, I think we can all agree that most of the reports produced so far (with the exception of the back-alteration case) are meaningful even without automatic enforcing of REQUIRED_USE, and fixing them would benefit our users already. I would like to ask you to look for the reports on your packages and fix them whenever possible. Feel free to ping me if you need any help with that.

Once the number of non-conforming packages goes down, I will convert the reports successively into warning levels, making the CI report new issues and the pull request scans proactively complain about them.

July 04, 2017
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
New autoconf, new problems (July 04, 2017, 18:03 UTC)

So a new release of autoconf is in the tree, it’s version 2.62. Guess what? It’s far from glitches-free. Although I suppose the biggest problem is from programs that bend the rules autoconf gives you. While the warnings it spews up during Amarok build are, well, just a warning, which will probably be fixed before a version of autoconf making those problems fatal, there are subtle bugs that might appear, like -bug #217154 which made PAM fail to build.

For a series of fortunate circumstances, since last week I’ve restarted packaging Ruby libraries, that I needed both for my own blog (Typo) and other stuff. As usual, I’m trying to avoid packaging gems since they can be quite of a pain in the bum to deal with. But since that’s something it’s known already I want to post a few more notes, especially in light of the fact we’ll hopefully be having a decent way to package gems in Gentoo by the end of the Summer.

Again gitarella (July 04, 2017, 18:03 UTC)

So, last night I couldn’t sleep, and I then decided to continue with what I do when I cannot sleep: code. I think this would be one of the last times that I have time to do this tho, but I’d wait the official results to say that. Now gitarella loads fine repositories in which there are merges (commits with more than one parent), as ruby-hunspell came to be, plus I’ve added the tag display à-la GitWeb, and added an option to rename the title of Gitarella pages.

No more envelopes! (July 04, 2017, 18:03 UTC)

And finally the job is complete! Tomorrow I’ll probably pass reading Gibson’s Neuromancer to relax, as I’m the tiredness personified. Now, after having to take a forced break yesterday to fix a nasty KDE regression with KSpell2 (you know, zander, ranting won’t bring you anywhere…), I’m working on the next ebuild for Amarok, 1.4.4. With this version, there is a partial Karma device support (as I wrote here); but I didn’t get much requests, and no devs seem to have a Rio Karma device (as far as I remember).

QA by disagreement (July 04, 2017, 18:03 UTC)

A few months ago I criticised Qt team for not following QA indications regarding the installation of documentation. Now, I wish to apologize and thank them: they were tremendously useful in identifying one area of Gentoo that needs fixes in both the QA policy and the actual use of it. The problem: the default QA rules, and the policies encoded in both the Ebuild HOWTO (that should be deprecated!) and Portage itself, is to install the documentation of packages into /usr/share/doc/${PF}, a path that changes between different revisions of the same package.

Blast from the recent past; two years ago (July 04, 2017, 18:03 UTC)

Today I was feeling somewhat blue, mostly because I’m demotivated to do most stuff, and I wanted to see what it was like to work in Gentoo two years ago. One thing I read is that a little shy of exactly two years ago, ICQ broke Kopete, just like they did yesterday. Interestingly enough, even though a workaround has been found for Kopete 0.12 (the one shipped with KDE 3.5), there is no bump I see in the tree this time.

Powered by.. Gentoo/FreeBSD (July 04, 2017, 18:03 UTC)

Okay, I want to thanks really, really much blackace for the “*Powered by Gentoo/FreeBSD*” logo I’ve put on the right on the blog :) I’m also trying to understand how to set a favicon with Gentoo/FreeBSD’s logo, but I’m not yet successful yet. Mostly because I suck at webmastering. This blog might remain a single experiment, a test, as Daniel (dsd) seems to be ready to update b2evolution on Planet Gentoo, so that it will improve antispam capabilities.

Some more notes about the setup I’m following for the vserver (where this blog and xine’s bugtracker’s run, the last one I think was similar, but about awstats. You can see the running thread: running stuff as user rather than root, especially if the user has no access to any particular resource. So this time my target is git-daemon; I already set it up some time ago to run as nobody (just changed the start-stop-daemon parameters) but there is some problem with that: you either give access to the git repositories to world, or to nobody, which might be used by other services too.

Back to ... KDE‽ (July 04, 2017, 18:03 UTC)

Now this is one of the things that you wouldn’t expect — Years after I left KDE behind me, today I’m back using it.. and honestly I feel again at home. I guess the whole backend changes that got through KDE4 were a bit too harsh for me at the time, and I could feel the rough edges. But after staying with GNOME2, XFCE, then Cinnamon.. this seems to be the best fit for me at this point.

Maintaining backports with GIT (July 04, 2017, 18:03 UTC)

I have written last week of the good feeling of merging patches upstream – even though since then I don’t think I got anything else merged … well, beside the bti fixes that I sent Greg – this week, let’s start with the opposite problem: how can you handle backports sanely, and have a quick way to check what was merged upstream? Well, the answer, at least for the software that is managed upstream with GIT, is quite easy to me.

June 27, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Open Source Summit Japan-2017 (June 27, 2017, 13:57 UTC)

Open Source Summit Japan 2017 summary

OSS Japan 2017 was a really great experience.

I sended my paper proposal and waited for a replay, some week after I got a
invite to partecipate at the Kernel Keynote.
I thought partecipating at the Kernel Keynote as mentor and doing a presentation
was a good way to talk about Gentoo Kernel Project and how to contribute in the
Linux Kernel and Gentoo Kernel Project.
Also my paper got accepted so I could join OSS Japan 2017 as speaker.
It was three really nice days.

Presentation:

Fast Releasing and Testing of Gentoo Kernel Packages and Future Plans of the Gentoo Kernel Project

My talk was manly about the Gentoo Kernel related Projects past and future
specifically about the Gentoo Kernel Continuos Integreting system we are creating:
https://github.com/gentoo/Gentoo_kernelCI

Why is needed:

  • We need some way for checking the linux-patches commits automatically, can also check pre-commit by pushing to a sandbox branch
  • Check the patches signatures
  • Checking the ebuild committed to https://github.com/gentoo/gentoo/commits/master/sys-kernel
  • Checking the kernel eclass commits
  • Checking the pull request to the sys-kernel/*
  • Use Qemu for testing kernel vmlinux correct execution

For any issue or contribution feel free to send here:
https://github.com/gentoo/Gentoo_kernelCI

For see Gentoo Kernel CI in action:
http://kernel1.amd64.dev.gentoo.org:8010

slides:
http://schd.ws/hosted_files/ossjapan2017/39/Gentoo%20Kernel%20recent%20and%20Future%20project.pdf

Open Source Summit Japan 2017
Keynote: Linux Kernel Panel - Moderated by Alice Ferrazzi, Gentoo Kernel Project Leader

The keynote was with:
Greg Kroah-Hartman - Fellow, Linux Foundation
Steven Rostedt - VMware
Dan Williams - Intel Open Source Technology Center
Alice Ferrazzi - Gentoo Kernel Project Leader, Gentoo

One interesting part was about how to contribute to the Linux Kernel.
After some information about Linux Kernel contribution numbers the talk moved on
ho to contribute in the Linux Kernel.
For contribute in the Linux Kernel there is need of some understanding of C
and running test in the Linux Kernel.
Like fuego, kselftest, coccinelle, and many others.
And also a good talk from Steven Rostedt about working with Real-Time patch.

Who can find the Gentoo logo in this image:

June 25, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Google-Summer-of-Code-summary week04 (June 25, 2017, 17:59 UTC)

Google Summer of Code summary week 04

What I did in this week 04 summary:

elivepatch:

  • Created the elivepatch client command line argument parser
  • Added function for sending patch and configuration files
  • Divided the call for sending (patch, config) and the call for building the livepatch
  • made send_file function more generic for sending all kind of files using RESTful api
  • Cleaned code following pep8
  • Decided to use only SSL and to don't use basic auth
  • Sending informations about the kernel version when requesting a livepatch build
  • We can now build livepatch using the RESTful API
  • Returning information about the livepatch building status

Kpatch:

  • Working on making kpatch-build working also with gentoo with all the features (As now kpatch-build can only automatically build livepatch for Ubuntu, Debian, Red Hat, Fedora)

Others:

  • Ask infra for a server for install the elivepatch server

What I need to do next time:

  • Finish the function for download the livepatch to the client
  • Testing elivepatch
  • Implementing the CVE patch uploader
  • Installing elivepatch to the Gentoo server
  • Fix kpatch-build for automatically work with gentoo-sources
  • Add more features to elivepatch

Google-Summer-of-Code-day18 (June 25, 2017, 17:59 UTC)

Google Summer of Code day 18

What was my plan for today?

  • going on with the code for retriving the livepatch and installing it

What i did today?

checked about kpatch-build required folder.

kpatch-build find_dirs function:

find_dirs() {
  if [[ -e "$SCRIPTDIR/create-diff-object" ]]; then
      # git repo
      TOOLSDIR="$SCRIPTDIR"
      DATADIR="$(readlink -f $SCRIPTDIR/../kmod)"
  elif [[ -e "$SCRIPTDIR/../libexec/kpatch/create-diff-object" ]]; then
      # installation path
      TOOLSDIR="$(readlink -f $SCRIPTDIR/../libexec/kpatch)"
      DATADIR="$(readlink -f $SCRIPTDIR/../share/kpatch)"
  else
      return 1
  fi
}

$SCRIPTDIR is the kpatch-build directory. kpatch-build is installed in /usr/bin/ so /usr/kmod /usr/libexe are all under such directory.

error "CONFIG_FUNCTION_TRACER, CONFIG_HAVE_FENTRY, CONFIG_MODULES, CONFIG_SYSFS, CONFIG_KALLSYMS_ALL kernel config options are required" Require by kmod/core.c: https://github.com/dynup/kpatch/blob/master/kmod/core/core.c#L62

We probably need someway for check that this setting are configured in the kernel we are going to build.

Updating kpatch-build for work automatically with gentoo (as now fedora for example can automatically download the kernel rpm and install it, we could do similar thing with gentoo): https://github.com/aliceinwire/kpatch/commits/gentoo

Starting to write the live patch downloader: https://github.com/aliceinwire/elivepatch/commit/d26611fb898223f2ea2dcf323078347ca928cbda

Now the elivepatch server can call and build the livepatch with kpatch:

sudo kpatch-build -s /usr/src/linux-4.10.14-gentoo/ -v /usr/src/linux-4.10.14-gentoo//vmlinux -c config 1.patch --skip-gcc-check
ERROR: kpatch build failed. Check /root/.kpatch/build.log for more details.
127.0.0.1 - - [25/Jun/2017 05:27:06] "POST /elivepatch/api/v1.0/build_livepatch HTTP/1.1" 201 -
WARNING: Skipping gcc version matching check (not recommended)
Using source directory at /usr/src/linux-4.10.14-gentoo
Testing patch file
checking file fs/exec.c
Hunk #1 succeeded at 259 (offset 16 lines).
Reading special section data
Building original kernel

Fixed some minor pep8

what i will do next time?
* work on the livepatch downloader and make the kpatch creator flexible

June 23, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Google-Summer-of-Code-day17 (June 23, 2017, 23:12 UTC)

Google Summer of Code day 16

What was my plan for today?

  • going on with the code for retriving the livepatch and installing it
  • Ask infra for a server where to install elivepatch sever

What i did today?
Sended request for the server that will offer the elivepatch service as talked with my mentor. https://bugs.gentoo.org/show_bug.cgi?id=622476

Fixed some pep8 warnings.

Livepatch server is now returning information about the livepatch building status.

Removed basic auth as we will go with SSL.

The client is now sending information about the kernel version when requesting a new build.

The kernel directory under the server is now a livepatch class variable.

what i will do next time?

  • going on with the code for retriving the livepatch and installing it

Google-Summer-of-Code-day16 (June 23, 2017, 23:12 UTC)

Google Summer of Code day 16

What was my plan for today?

  • Divide call for sending (patch, config) and the call for build the livepatch
  • Make the livepatch call more flexible (as now is hardcoded)
  • Ask infra for a server where to install elivepatch sever

What i did today?

Added patch file path argument to the elivepatch server API and added patch call to elivepatch client.

Adding way for dividing the call for sending the configuration with a POST call sending the patch with a POST calland than start the livepatch build and getting the result.

patch sended work and working on calling livepatch.

Added docstring to the build patch function.

Cleaned GetLive dispatcher function.

Added call from client to build livepatch of the server API.

made send_file function more generic for send all kind of file.

what i will do next time?

  • going on with the code for retriving the livepatch and installing it

June 21, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
2017-06-21 Google-Summer-of-Code-day14 (June 21, 2017, 17:57 UTC)

Google Summer of Code day 15

What was my plan for today?
working on sending the configuration file on RESTful api,
and starting to work on making the patch.ko file in the server.

What i did today?
using wekzeug.datastructures.FileStorage in elivepatch_server,
I could receive the file from the elivepatch_client POST request
using the RESTful API.

def post(self):
    parse = reqparse.RequestParser()
    parse.add_argument('file', type=werkzeug.datastructures.FileStorage, location)

so as now we can get the kernel configuration file, extract if is .gz filename and send it to the elivepatch server.

elivepatch server need to read the configuration, compare it with the
current kernel configuration and if different recompile the kernel.
After we can start making the livepatch with kpatch-build.

This is the example of using kpatch-build:

kpatch-build/kpatch-build -s /usr/src/linux-4.9.16-gentoo/ -v /usr/src/linux-4.9.16-gentoo/vmlinux examples/test.patch --skip-gcc-check
gsoc-2017 kpatch (gentoo) # kpatch-build/kpatch-build --help
usage: kpatch-build [options] <patch file>
            -h, --help         Show this help message
            -r, --sourcerpm    Specify kernel source RPM
            -s, --sourcedir    Specify kernel source directory
            -c, --config       Specify kernel config file
            -v, --vmlinux      Specify original vmlinux
            -t, --target       Specify custom kernel build targets
            -d, --debug        Keep scratch files in /tmp
            --skip-cleanup     Skip post-build cleanup
            --skip-gcc-check   Skip gcc version matching check
                               (not recommended)

This command is called automatically by the elivepatch server after receiving the configuration file.

we need also to send the patch file.

what i will do next time?

  • Divide call for sending (patch, config) and the call for build the livepatch
  • Make the livepatch call more flexible (as now is hardcoded)
  • Ask infra for a server where to install elivepatch sever

June 16, 2017
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Recently, Gentoo Linux put GCC 6.3 (released in December 2016) into the testing branch. For a source-based distribution like Gentoo, GCC is a critical part of the toolchain, and sometimes can lead to packages failing to compile or run. I recently ran into just this problem with Audacity. The error that I hit was not during compilation, but during runtime:


$ audacity
Fatal Error: Mismatch between the program and library build versions detected.
The library used 3.0 (wchar_t,compiler with C++ ABI 1009,wx containers,compatible with 2.8),
and your program used 3.0 (wchar_t,compiler with C++ ABI 1010,wx containers,compatible with 2.8).
Aborted

The Gentoo Wiki has a nice, detailed page on Upgrading GCC, and explicitly calls out ABI changes. In this particular case of Audacity, the problematic library is referenced in the error above: “wx containers”. WX containers are handled by the wxGTK package. So, I simply needed to rebuild the currently-installed version of wxGTK to fix this particular problem.

Hope that helps.

Cheers,
Zach

June 15, 2017
Hanno Böck a.k.a. hanno (homepage, bugs)
Don't leave Coredumps on Web Servers (June 15, 2017, 09:20 UTC)

CoreCoredumps are a feature of Linux and other Unix systems to analyze crashing software. If a software crashes, for example due to an invalid memory access, the operating system can save the current content of the application's memory to a file. By default it is simply called core.

While this is useful for debugging purposes it can produce a security risk. If a web application crashes the coredump may simply end up in the web server's root folder. Given that its file name is known an attacker can simply download it via an URL of the form https://example.org/core. As coredumps contain an application's memory they may expose secret information. A very typical example would be passwords.

PHP used to crash relatively often. Recently a lot of these crash bugs have been fixed, in part because PHP now has a bug bounty program. But there are still situations in which PHP crashes. Some of them likely won't be fixed.

How to disclose?

With a scan of the Alexa Top 1 Million domains for exposed core dumps I found around 1.000 vulnerable hosts. I was faced with a challenge: How can I properly disclose this? It is obvious that I wouldn't write hundreds of manual mails. So I needed an automated way to contact the site owners.

Abusix runs a service where you can query the abuse contacts of IP addresses via a DNS query. This turned out to be very useful for this purpose. One could also imagine contacting domain owners directly, but that's not very practical. The domain whois databases have rate limits and don't always expose contact mail addresses in a machine readable way.

Using the abuse contacts doesn't reach all of the affected host operators. Some abuse contacts were nonexistent mail addresses, others didn't have abuse contacts at all. I also got all kinds of automated replies, some of them asking me to fill out forms or do other things, otherwise my message wouldn't be read. Due to the scale I ignored those. I feel that if people make it hard for me to inform them about security problems that's not my responsibility.

I took away two things that I changed in a second batch of disclosures. Some abuse contacts seem to automatically search for IP addresses in the abuse mails. I originally only included affected URLs. So I changed that to include the affected IPs as well.

In many cases I was informed that the affected hosts are not owned by the company I contacted, but by a customer. Some of them asked me if they're allowed to forward the message to them. I thought that would be obvious, but I made it explicit now. Some of them asked me that I contact their customers, which again, of course, is impractical at scale. And sorry: They are your customers, not mine.

How to fix and prevent it?

If you have a coredump on your web host, the obvious fix is to remove it from there. However you obviously also want to prevent this from happening again.

There are two settings that impact coredump creation: A limits setting, configurable via /etc/security/limits.conf and ulimit and a sysctl interface that can be found under /proc/sys/kernel/core_pattern.

The limits setting is a size limit for coredumps. If it is set to zero then no core dumps are created. To set this as the default you can add something like this to your limits.conf:

* soft core 0

The sysctl interface sets a pattern for the file name and can also contain a path. You can set it to something like this:

/var/log/core/core.%e.%p.%h.%t

This would store all coredumps under /var/log/core/ and add the executable name, process id, host name and timestamp to the filename. The directory needs to be writable by all users, you should use a directory with the sticky bit (chmod +t).

If you set this via the proc file interface it will only be temporary until the next reboot. To set this permanently you can add it to /etc/sysctl.conf:

kernel.core_pattern = /var/log/core/core.%e.%p.%h.%t

Some Linux distributions directly forward core dumps to crash analysis tools. This can be done by prefixing the pattern with a pipe (|). These tools like apport from Ubuntu or abrt from Fedora have also been the source of security vulnerabilities in the past. However that's a separate issue.

Look out for coredumps

My scans showed that this is a relatively common issue. Among popular web pages around one in a thousand were affected before my disclosure attempts. I recommend that pentesters and developers of security scan tools consider checking for this. It's simple: Just try download the /core file and check if it looks like an executable. In most cases it will be an ELF file, however sometimes it may be a Mach-O (OS X) or an a.out file (very old Linux and Unix systems).

Image credit: NASA/JPL-Université Paris Diderot

June 13, 2017
Alice Ferrazzi a.k.a. alicef (homepage, bugs)
Moving back to ikiwiki (June 13, 2017, 03:40 UTC)

I was trying to use blogs.gentoo.org/alicef/ Gentoo official blog
based on wordpress.
As far as I could like the draft feature, it had some big drawback.
Most big one I couldn't post any syntax highlighted code.
And wordpress maintenance takes lots of time, in particular managing plugins.
Also because I cannot change Gentoo blog plugins without admin privilege, is bit too much to have to ask every time I have problem with plugins or I need a new one.

So I decided to come back to ikiwiki.
Also in ikiwiki I can make some sort of draft function where post tagged as draft dosen't come up on the blog list.
This is simply done by using this on the blog.mdwn
pages="page(blog/) and !/Discussion and !*/local.css and !tagged(draft)"
That will remove the blog page tagged with draft from the blog view.

For the syntax highlight I used a plugin for pygments.py made by
tylercipriani.com
That you can find here pygments.pm

Pygments.pm output example:

#include <stdio.h>

int main(void) {
    // your code goes here
    return 0;
}

And last one, on Ikiwiki comments, I decided to delegate comments spam to disqus, also if I don't like so much to use a private business plugin, do is job of managing comments well enough.
I also just discovered that disqus is part of Y combinator, the company behind hacker news.

[2017 05] Gentoo activity summary (June 13, 2017, 01:49 UTC)

[2017 04] Gentoo activity summary (June 13, 2017, 01:49 UTC)

  • Searching work as researcher in Japan
  • Got instances for Gentoo Kernel CI
  • Joined Gentoo Kernel Security Project
  • Released kernel 4.11

[2017 03] Gentoo activity summary (June 13, 2017, 01:43 UTC)

June 10, 2017
Matthew Thode a.k.a. prometheanfire (homepage, bugs)
Gentoo Puppet Portage Package Provider (June 10, 2017, 05:00 UTC)

Why do this

The previus built in puppet portage package provider (I'm just going to shorten it to PPPP) only supported very simplistic package interactions. Mainly package name (with slot) install and uninstall. This has proven fairly limiting, if you want to install a specific version of a package and lock it down you were forced to call out to exec or editing package.{mask,unmask,keywords} files.

The new provider (which will be built into puppet in 5.0 or puppet-agent-2.0) supports all the package provider attributes.

How do I get this awesome thing

Emerge puppet or puppet-agent with the experimental use flag.

What it can do

You can use the following attributes with the new PPPP.

  • Name - The full package atom works now, using qatom on the backend.
  • ensure - now allowing a package purge as well (CONFIG_PROTECT="-*").
  • install_options - you can now pass options to emerge (--deep or --usepkgonly for example).
  • uninstall_options - just like install_options

Being able to call out specific versions and per package install options will give much greater flexability.

fin

Here is the pull request that upstream puppet merged.

If you have any questions I'm on freenode as prometheanfire.

June 03, 2017
Matthew Thode a.k.a. prometheanfire (homepage, bugs)
Gentoo portage templates (June 03, 2017, 05:00 UTC)

Why do this

Gentoo is known for being somewhat complex to manage, making clusters of gentoo machines even more complex in most scenarios. Using the following methods the configuration becomes easier.

By the end of this you should be able to have a default hiera configuration for Gentoo while still being able to override it for specific use cases. What makes the method I chose particularly powerful is the ability to delete default vales entirely, not just setting them to something else.

Most of these methods came from my experience with chef that I thought would apply well to other config engines. While some don't like shoving logic to the configuration template engine, I'm open to suggestions.

Requirements

  • Puppet 4.x or puppet-agent with hiera support.
  • Puppet's stdlib installed (specifically for delete_values stuff).
  • (optional) use puppetserver instead of running this oneoff.
  • Hiera configured to use the following configuration.

Hiera config

:merge_behavior: deeper
:deep_merge_options:
  :merge_hash_arrays: true

Basic Setup

  • Convert the common portage configuraitons to templates.
  • Convert the data in those templates to a datastructure.
  • Use hiera to write the defaults / node overrides.
  • Call the templates via a puppet module.

Datastructures

The easiest way of explaing how this works is to say that the only data stored in the 'deepest' value is ever going to be True or False. The reason for this is a because using deep_merge in hiera is an additive process and we need a flag to remove things further down the line.

The datastructure itself is fairly simple, here is an excerpt from my setup.

make_conf:
  emerge_default_opts:
    "--quiet-build": true
    "--changed-use": true

If I wanted to disable --quiet-build down the line you would just set the value to False in a higher precidence (the specific node config instead of the general location.

make_conf:
  emerge_default_opts:
    "--quiet-build": false

Configuration Templates

The templates themselves are epp based, not erb (the old method).

package.keywords

For this one I'll also supply how I auto-set the right archetecture, works for amd64 at least.

Hiera data

"app-admin/paxtest ~%{facts.architecture}": true

Template

<%- |$packages| -%>
# THIS FILE WAS GENERATED BY PUPPET, CHANGES WILL BE OVERWRITTEN

<%- keys(delete_values($packages, false)).each |$package| { -%>
<%= "$package" %>
<%- } -%>

This one is the simplest, if a value for the key (paxtest in this case) is set to false, don't use it, the remaining keys are then set as plan text.

package.use

<%- |$packages| -%>
# THIS FILE WAS GENERATED BY PUPPET, CHANGES WILL BE OVERWRITTEN

<%- keys(delete_values($packages, false)).each |$package| { -%>
  <%- if ! empty(keys(delete_values($packages[$package], false))) { -%>
<%= "$package" %> <%= join(keys(delete_values($packages[$package], false)), ' ') %>
  <%- } -%>
<%- } -%>

This one is fairly straight forward as well, for each package that isn't disabled, if there are keys for the package (signifying use flags, needed because we remove the unset flags) then set them. This combines the flags set from all levels in hiera.

make.conf

This will be the most complicated one, but it's also likely to be the most important. I'll explain a bit about it after the paste.

<%- |$config| -%>
# THIS FILE WAS GENERATED BY PUPPET, CHANGES WILL BE OVERWRITTEN

CFLAGS="<%= join(keys(delete_values($config['cflags'], false)), ' ') %>"
CXXFLAGS="<%= join(keys(delete_values($config['cxxflags'], false)), ' ') %>"
CHOST="<%= $config['chost'] %>"
MAKEOPTS="<%= join(keys(delete_values($config['makeopts'], false)), ' ') %>"
CPU_FLAGS_X86="<%= join(keys(delete_values($config['cpu_flags_x86'], false)), ' ') %>"
ABI_X86="<%= join(keys(delete_values($config['abi_x86'], false)), ' ') %>"

USE="<%= join(keys(delete_values($config['use'], false)), ' ') %>"

GENTOO_MIRRORS="<%= join(keys(delete_values($config['gentoo_mirrors'], false)), ' ') %>"
<% if has_key($config, 'portage_binhost') { -%>
  <%- if $config['portage_binhost'] != false { -%>
PORTAGE_BINHOST="<%= $config['portage_binhost'] %>"
  <%- } -%>
<% } -%>

FEATURES="<%= join(keys(delete_values($config['features'], false)), ' ') %>"
EMERGE_DEFAULT_OPTS="<%= join(keys(delete_values($config['emerge_default_opts'], false)), ' ') %>"
PKGDIR="<%= $config['pkgdir'] %>"
PORT_LOGDIR="<%= $config['port_logdir'] %>"
PORTAGE_GPG_DIR="<%= $config['portage_gpg_dir'] %>"
PORTAGE_GPG_KEY='<%= $config['portage_gpg_key'] %>'

GRUB_PLATFORMS="<%= join(keys(delete_values($config['grub_platforms'], false)), ' ') %>"
LINGUAS="<%= join(keys(delete_values($config['linguas'], false)), ' ') %>"
L10N="<%= join(keys(delete_values($config['l10n'], false)), ' ') %>"

PORTAGE_ELOG_CLASSES="<%= join(keys(delete_values($config['portage_elog_classes'], false)), ' ') %>"
PORTAGE_ELOG_SYSTEM="<%= join(keys(delete_values($config['portage_elog_system'], false)), ' ') %>"
PORTAGE_ELOG_MAILURI="<%= $config['portage_elog_mailuri'] %>"
PORTAGE_ELOG_MAILFROM="<%= $config['portage_elog_mailfrom'] %>"

<% if has_key($config, 'accept_licence') { -%>
ACCEPT_LICENSE="<%= join(keys(delete_values($config['accept_licence'], false)), ' ') %>"
<%- } -%>
POLICY_TYPES="<%= join(keys(delete_values($config['policy_types'], false)), ' ') %>"
PAX_MARKINGS="<%= join(keys(delete_values($config['pax_markings'], false)), ' ') %>"

USE_PYTHON="<%= join(keys(delete_values($config['use_python'], false)), ' ') %>"
PYTHON_TARGETS="<%= join(keys(delete_values($config['python_targets'], false)), ' ') %>"
RUBY_TARGETS="<%= join(keys(delete_values($config['ruby_targets'], false)), ' ') %>"
PHP_TARGETS="<%= join(keys(delete_values($config['php_targets'], false)), ' ') %>"

<% if has_key($config, 'source') { -%>
source <%= join(keys(delete_values($config['source'], false)), ' ') %>
<%- } -%>

The basic idea of this is that you pass in the full make.conf datastructre you will generate as a single variable. Everything else is pulled (or elemated from that).

Each option that is selected already has all the options merged, but this could mean both the disabled versions of a given value could be still there, this is removed using the delete_values($config['foo'], false) bit.

The puppet module itself

It's fairly easy to call, just make sure the template is in the template location and do it as follows.

file { '/etc/portage/make.conf':
  content => epp('portage/portage-make_conf.epp', {'config' => hiera_hash('portage')['make_conf']})
}

fin

If you have any questions I'm on freenode as prometheanfire.

May 16, 2017
Alexys Jacob a.k.a. ultrabug (homepage, bugs)

In my previous blog post, I demonstrated how to use the PIV feature of a Yubikey to add a 2nd factor authentication to SSH.

Careful readers such as Grzegorz Kulewski pointed out that using the GPG capability of the Yubikey was also a great, more versatile and more secure option on the table (I love those community insights):

  • GPG keys and subkeys are indeed more flexible and can be used for case-specific operations (signing, encryption, authentication)
  • GPG is more widely used and one could use their Yubikey smartcard for SSH, VPN, HTTP auth and code signing
  • The Yubikey 4 GPG feature supports 4096 bit keys (limited to 2048 for PIV)

While I initially looked at the GPG feature, its apparent complexity got me to discard it for my direct use case (SSH). But I couldn’t resist the good points of Grzegorz and here I got back into testing it. Thank you again Grzegorz for the excuse you provided 😉

So let’s get through with the GPG feature of the Yubikey to authenticate our SSH connections. Just like the PIV method, this one has the  advantage to allow a 2nd factor authentication while using the public key authentication mechanism of OpenSSH and thus does not need any kind of setup on the servers.

Method 3 – SSH using Yubikey and GPG

Acknowledgement

The first choice you have to make is to decide whether you allow your master key to be stored on the Yubikey or not. This choice will be guided by how you plan to use and industrialize your usage of the GPG based SSH authentication.

Consider this to choose whether to store the master key on the Yubikey or not:

  • (con) it will not allow the usage of the same GPG key on multiple Yubikeys
  • (con) if you loose your Yubikey, you will have to revoke your entire GPG key and start from scratch (since the secret key is stored on the Yubikey)
  • (pro) by storing everything on the Yubikey, you won’t necessary have to have an offline copy of your master key (and all the process that comes with it)
  • (pro) it is easier to generate and store everything on the key and is then a good starting point for new comers or rare GPG users

Because I want to demonstrate and enforce the most straightforward way of using it, I will base this article on generating and storing everything on a Yubikey 4. You can find useful links at the end of the article pointing to reference on how to do it differently.

Tools installation

For this to work, we will need some tools on our local machine to setup our Yubikey correctly.

Gentoo users should install those packages:

emerge -av dev-libs/opensc sys-auth/ykpers app-crypt/ccid sys-apps/pcsc-tools app-crypt/gnupg

Gentoo users should also allow the pcscd service to be hotplugged (started automatically upon key insertion) by modifying their /etc/rc.conf and having:

rc_hotplug="pcscd"

Yubikey setup

The idea behind the Yubikey setup is to generate and store the GPG keys directly on our Yubikey and to secure them via a PIN code (and an admin PIN code).

  • default PIN code: 123456
  • default admin PIN code: 12345678

First, insert your Yubikey and let’s change its USB operating mode to OTP+U2F+CCID with MODE_FLAG_EJECT flag.

ykpersonalize -m86
Firmware version 4.3.4 Touch level 783 Program sequence 3

The USB mode will be set to: 0x86

Commit? (y/n) [n]: y

NOTE: if you have an older version of Yubikey (before Sept. 2014), use -m82 instead.

Then, we can generate a new GPG key on the Yubikey. Let’s open the smartcard for edition.

gpg --card-edit --expert

Reader ...........: Yubico Yubikey 4 OTP U2F CCID (0005435106) 00 00
Application ID ...: A7560001240102010006054351060000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 75435106
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

Then switch to admin mode.

gpg/card> admin
Admin commands are allowed

We can start generating the Signature, Encryption and Authentication keys on the Yubikey. During the process, you will be prompted alternatively for the admin PIN and PIN.

gpg/card> generate 
Make off-card backup of encryption key? (Y/n) 

Please note that the factory settings of the PINs are
   PIN = '123456'     Admin PIN = '12345678'
You should change them using the command --change-pin

I advise you say Yes to the off-card backup of the encryption key.

Yubikey 4 users can choose a 4096 bits key, let’s go for it for every key type.

What keysize do you want for the Signature key? (2048) 4096
The card will now be re-configured to generate a key of 4096 bits
Note: There is no guarantee that the card supports the requested size.
      If the key generation does not succeed, please check the
      documentation of your card to see what sizes are allowed.
What keysize do you want for the Encryption key? (2048) 4096
The card will now be re-configured to generate a key of 4096 bits
What keysize do you want for the Authentication key? (2048) 4096
The card will now be re-configured to generate a key of 4096 bits

Then you’re asked for the expiration of your key. I choose 1 year but it’s up to you (leave 0 for no expiration).

Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at mer. 15 mai 2018 21:42:42 CEST
Is this correct? (y/N) y

Finally you give GnuPG details about your user ID and you will be prompted for a passphrase (make it strong).

GnuPG needs to construct a user ID to identify your key.

Real name: Ultrabug
Email address: ultrabug@nospam.com
Comment: 
You selected this USER-ID:
    "Ultrabug <ultrabug@nospam.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

If you chose to make an off-card backup of your key, you will also get notified of its location as well the revocation certificate.

gpg: Note: backup of card key saved to '/home/ultrabug/.gnupg/sk_8E407636C9C32C38.gpg'
gpg: key 22A73AED8E766F01 marked as ultimately trusted
gpg: revocation certificate stored as '/home/ultrabug/.gnupg/openpgp-revocs.d/A1580FD98C0486D94C1BE63B22A73AED8E766F01.rev'
public and secret key created and signed.

Make sure to store that backup in a secure and offline location.

You can verify that everything went good and take this chance to note the public key ID.

gpg/card> verify

Reader ...........: Yubico Yubikey 4 OTP U2F CCID (0001435106) 00 00
Application ID ...: A7560001240102010006054351060000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 75435106
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 4
Signature key ....: A158 0FD9 8C04 86D9 4C1B E63B 22A7 3AED 8E76 6F01
 created ....: 2017-05-16 20:43:17
Encryption key....: E1B6 7009 907D 1D94 B200 37D7 8E40 7636 C9C3 2C38
 created ....: 2017-05-16 20:43:17
Authentication key: AAED AB8E E055 41B2 EFFF 62A4 164F 873A 75D2 AD6B
 created ....: 2017-05-16 20:43:17
General key info..: pub rsa4096/22A73AED8E766F01 2017-05-16 Ultrabug <ultrabug@nospam.com>
sec> rsa4096/22A73AED8E766F01 created: 2017-05-16 expires: 2018-05-16
 card-no: 0001 05435106
ssb> rsa4096/164F873A75D2AD6B created: 2017-05-16 expires: 2018-05-16
 card-no: 0001 05435106
ssb> rsa4096/8E407636C9C32C38 created: 2017-05-16 expires: 2018-05-16
 card-no: 0001 05435106

You’ll find the public key ID on the “General key info” line (22A73AED8E766F01):

General key info..: pub rsa4096/22A73AED8E766F01 2017-05-16 Ultrabug <ultrabug@nospam.com>

Quit the card edition.

gpg/card> quit

It is then convenient to upload your public key to a key server, whether public or on your own web server (you can also keep it to be used and imported directly from an USB stick).

Export the public key:

gpg --armor --export 22A73AED8E766F01 > 22A73AED8E766F01.asc

Then upload it to your http server or a public server (needed if you want to be able to easily use the key on multiple machines):

# Upload it to your http server
scp 22A73AED8E766F01.asc user@server:public_html/static/22A73AED8E766F01.asc

# OR upload it to a public keyserver
gpg --keyserver hkps://hkps.pool.sks-keyservers.net --send-key 22A73AED8E766F01

Now we can finish up the Yubikey setup. Let’s edit the card again:

gpg --card-edit --expert

Reader ...........: Yubico Yubikey 4 OTP U2F CCID (0001435106) 00 00
Application ID ...: A7560001240102010006054351060000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 75435106
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 4
Signature key ....: A158 0FD9 8C04 86D9 4C1B E63B 22A7 3AED 8E76 6F01
 created ....: 2017-05-16 20:43:17
Encryption key....: E1B6 7009 907D 1D94 B200 37D7 8E40 7636 C9C3 2C38
 created ....: 2017-05-16 20:43:17
Authentication key: AAED AB8E E055 41B2 EFFF 62A4 164F 873A 75D2 AD6B
 created ....: 2017-05-16 20:43:17
General key info..: pub rsa4096/22A73AED8E766F01 2017-05-16 Ultrabug <ultrabug@nospam.com>
sec> rsa4096/22A73AED8E766F01 created: 2017-05-16 expires: 2018-05-16
 card-no: 0001 05435106
ssb> rsa4096/164F873A75D2AD6B created: 2017-05-16 expires: 2018-05-16
 card-no: 0001 05435106
ssb> rsa4096/8E407636C9C32C38 created: 2017-05-16 expires: 2018-05-16
 card-no: 0001 05435106
gpg/card> admin

Make sure that the Signature PIN is forced to request that your PIN is entered when your key is used. If it is listed as “not forced”, you can enforce it by entering the following command:

gpg/card> forcesig

It is also good practice to set a few more settings on your key.

gpg/card> login
Login data (account name): ultrabug

gpg/card> lang
Language preferences: en

gpg/card> name 
Cardholder's surname: Bug
Cardholder's given name: Ultra

Now we need to setup the PIN and admin PIN on the card.

gpg/card> passwd 
gpg: OpenPGP card no. A7560001240102010006054351060000 detected

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 1
PIN changed.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 3
PIN changed.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? Q

If you uploaded your public key on your web server or a public server, configure it on the key:

gpg/card> url
URL to retrieve public key: http://ultrabug.fr/keyserver/22A73AED8E766F01.asc

gpg/card> quit

Now we can quit the gpg card edition, we’re done on the Yubikey side!

gpg/card> quit

SSH client setup

This is the setup on the machine(s) where you will be using the GPG key. The idea is to import your key from the card to your local keyring so you can use it on gpg-agent (and its ssh support).

You can skip the fetch/import part below if you generated the key on the same machine than you are using it. You should see it listed when executing gpg -k.

Plug-in your Yubikey and load the smartcard.

gpg --card-edit --expert

Then fetch the key from the URL to import it to your local keyring.

gpg/card> fetch

Then you’re done on this part, exit gpg and update/display& your card status.

gpg/card> quit

gpg --card-status

You can verify the presence of the key in your keyring:

gpg -K
sec>  rsa4096 2017-05-16 [SC] [expires: 2018-05-16]
      A1580FD98C0486D94C1BE63B22A73AED8E766F01
      Card serial no. = 0001 05435106
uid           [ultimate] Ultrabug <ultrabug@nospam.com>
ssb>  rsa4096 2017-05-16 [A] [expires: 2018-05-16]
ssb>  rsa4096 2017-05-16 [E] [expires: 2018-05-16]

Note the “Card serial no.” showing that the key is actually stored on a smartcard.

Now we need to configure gpg-agent to enable ssh support, edit your ~/.gnupg/gpg-agent.conf configuration file and make sure that the enable-ssh-support is present:

default-cache-ttl 7200
max-cache-ttl 86400
enable-ssh-support

Then you will need to update your ~/.bashrc file to automatically start gpg-agent and override ssh-agent’s environment variables. Add this at the end of your ~/.bashrc file (or equivalent).

# start gpg-agent if it's not running
# then override SSH authentication socket to use gpg-agent
pgrep -l gpg-agent &>/dev/null
if [[ "$?" != "0" ]]; then
 gpg-agent --daemon &>/dev/null
fi
SSH_AUTH_SOCK=/run/user/$(id -u)/gnupg/S.gpg-agent.ssh
export SSH_AUTH_SOCK

To simulate a clean slate, unplug your card then kill any running gpg-agent:

killall gpg-agent

Then plug back your card and source your ~/.bashrc file:

source ~/.bashrc

Your GPG key is now listed in you ssh identities!

ssh-add -l
4096 SHA256:a4vsJM6Sw1Rt8orvPnI8nvNUwHbRQ67ylnoTxruozK9 cardno:000105435106 (RSA)

You will now be able to get the SSH public key hash to copy to your remote servers using:

ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCVDq24Ld/bOzc3yNnY6fF7FNfZnb6wRVdN2xMo1YiA5pz20y+2P1ynq0rb6l/wsSip0Owq4G6fzaJtT1pBUAkEJbuMvZBrbYokb2mZ78iFZyzAkCT+C9YQtvPClFxSnqSL38jBpunZuuiFfejM842dWdMNK3BTcjeTQdTbmY+VsVOw7ppepRh7BWslb52cVpg+bHhB/4H0BUUd/KHZ5sor0z6e1OkqIV8UTiLY2laCCL8iWepZcE6n7MH9KItqzX2I9HVIuLCzhIib35IRp6d3Whhw3hXlit4/irqkIw0g7s9jC8OwybEqXiaeQbmosLromY3X6H8++uLnk7eg9RtCwcWfDq0Qg2uNTEarVGgQ1HXFr8WsjWBIneC8iG09idnwZNbfV3ocY5+l1REZZDobo2BbhSZiS7hKRxzoSiwRvlWh9GuIv8RNCDss9yNFxNUiEIi7lyskSgQl3J8znDXHfNiOAA2X5kVH0s6AQx4hQP9Dl1X2Em4zOz+yJEPFnAvE+XvBys1yuUPq1c3WKMWzongZi8JNu51Yfj7Trm74hoFRn+CREUNpELD9JignxlvkoKAJpWVLdEu1bxJ7jh7kcMQfVEflLbfkEPLV4nZS4sC1FJR88DZwQvOudyS69wLrF3azC1Gc/fTgBiXVVQwuAXE7vozZk+K4hdrGq4u7Gw== cardno:000105435106

This is what ends up in ~/.ssh/authorized_keys on your servers.

When connecting to your remote server, you will be prompted for the PIN!

Conclusion

Using the GPG feature of your Yubikey is very convenient and versatile. Even if it is not that hard after all, it is interesting and fair to note that the PIV method is indeed more simple to implement.

When you need to maintain a large number of security keys in an organization and that their usage is limited to SSH, you will be inclined to stick with PIV if 2048 bits keys are acceptable for you.

However, for power users and developers, usage of GPG is definitely something you need to consider for its versatility and enhanced security.

Useful links

You may find those articles useful to setup your GPG key differently and avoid having the secret key tied to your Yubikey.

Sebastian Pipping a.k.a. sping (homepage, bugs)

Hi!

When I started fetchcommandwrapper about 6 years ago it was a proof of concept: It plugged into portage replacing wget for downloads, facilitating ${GENTOO_MIRRORS} and aria2 to both download faster and distribute loads across mirrors. A hack for sure, but with some potential.

Back then public interest was non-existent, fetchcommandwrapper had some issues — e.g. metadata.xsd downloads failed and some sites rejected downloading before it made aria2 dress like wget — and I stopped using it myself, eventually.

With the latest bug reports, bugfixes and release of version 0.8 in Gentoo, fetchcommandwrapper is ready for general use now. To give it a shot, you emerge app-portage/fetchcommandwrapper and append source /usr/share/fetchcommandwrapper/make.conf to /etc/portage/make.conf. Done.

If you have extra options that you would like to pass to aria2c, put them in ${FETCHCOMMANDWRAPPER_EXTRA}, or ${FETCHCOMMANDWRAPPER_OPTIONS} for fetchcommendwrapper itself; for example

FETCHCOMMANDWRAPPER_OPTIONS="--link-speed=600000"

tells fetchcommandwrapper that my download link has 600KB/s only and makes aria2 in turn drop connections to mirrors that cannot keep up with at least a third of that, so that faster mirrors get a chance to take their place.

For non-ebuild bugs, feel free to use https://github.com/gentoo/fetchcommandwrapper/issues to report.

Best, Sebastian

May 12, 2017
Alexys Jacob a.k.a. ultrabug (homepage, bugs)

In my previous blog post, I demonstrated how to use a Yubikey to add a 2nd factor (2FA) authentication to SSH using pam_ssh and pam_yubico.

In this article, I will go further and demonstrate another method using Yubikey’s Personal Identity Verification (PIV) capability.

This one has the huge advantage to allow a 2nd factor authentication while using the public key authentication mechanism of OpenSSH and thus does not need any kind of setup on the servers.

Method 2 – SSH using Yubikey and PIV

Yubikey 4 and NEO also act as smartcards supporting the PIV standard which allows you to store a private key on your security key through PKCS#11. This is an amazing feature which is also very good for our use case.

Tools installation

For this to work, we will need some tools on our local machines to setup our Yubikey correctly.

Gentoo users should install those packages:

emerge dev-libs/opensc sys-auth/ykpers sys-auth/yubico-piv-tool sys-apps/pcsc-lite app-crypt/ccid sys-apps/pcsc-tools sys-auth/yubikey-personalization-gui

Gentoo users should also allow the pcscd service to be hotplugged (started automatically upon key insertion) by modifying their /etc/rc.conf and having:

rc_hotplug="pcscd"

Yubikey setup

The idea behind the Yubikey setup is to generate and store a private key in our Yubikey and to secure it via a PIN code.

First, insert your Yubikey and let’s change its USB operating mode to OTP+CCID.

ykpersonalize -m2
Firmware version 4.3.4 Touch level 783 Program sequence 3

The USB mode will be set to: 0x2

Commit? (y/n) [n]: y

Then, we will create a new management key:

key=`dd if=/dev/random bs=1 count=24 2>/dev/null | hexdump -v -e '/1 "%02X"'`
echo $key
D59E46FE263DDC052A409C68EB71941D8DD0C5915B7C143A

Replace the default management key (if prompted, copy/paste the key printed above):

yubico-piv-tool -a set-mgm-key -n $key --key 010203040506070801020304050607080102030405060708

Then change the default PIN code and PUK code of your Yubikey

yubico-piv-tool -a change-pin -P 123456 -N <NEW PIN>

yubico-piv-tool -a change-puk -P 12345678 -N <NEW PUK>

Now that your Yubikey is secure, let’s proceed with the PCKS#11 certificate generation. You will be prompted for your management key that you generated before.

yubico-piv-tool -s 9a -a generate -o public.pem -k

Then create a self-signed certificate (only used for libpcks11) and import it in the Yubikey:

yubico-piv-tool -a verify-pin -a selfsign-certificate -s 9a -S "/CN=SSH key/" -i public.pem -o cert.pem
yubico-piv-tool -a import-certificate -s 9a -i cert.pem

Here you are! You can now export your public key to use with OpenSSH:

ssh-keygen -D opensc-pkcs11.so -e
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWtqI37jwxYMJ9XLq9VwHgJlhZViPVAGIUfMm8SAlfs6cka4Cj570lkoGK04r8JAVJFy/iKfhGpL9N9XuartfIoq6Cg/6Qvg3REupuqs51V2cBaC/gnWIQ7qZqlzBulvcOvzNfHFD/lX42J58+E8tWnYg6GzIsImFZQVpmI6SxNfSmVQIqxIufInrbQaI+pKXntdTQC9wyNK5FAA8TXAdff5ZDnmetsOTVble9Ia5m6gqM7MnxNZ56uDpn+6lCxRZSW+Ln2PDE7sivVcST4qpfwY4P4Lrb3vrjCGODFg4xmGNKXsLi2+uZbs5rW7bg4HFO50kKDucPV1M+rBWA9999

Copy to your servers your SSH public key to your usual ~/.ssh/authorized_keys file in your $HOME.

Testing PIV secured SSH

Plug-in your Yubikey, and then SSH to your remote server using the opensc-pkcs11 library. You will be prompted for your PIN and then successfully logged in 🙂

ssh -I opensc-pkcs11.so cheetah
Enter PIN for 'PIV_II (PIV Card Holder pin)':

You can then configure SSH to use it by default for all your hosts in your ~/.ssh/config

Host=*
PKCS11Provider /usr/lib/opensc-pkcs11.so

Using PIV with ssh-agent

You can also use ssh-agent to avoid typing your PIN every time.

When asked for the passphrase, enter your PIN:

ssh-add -s /usr/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/lib/opensc-pkcs11.so

You can verify that it worked by listing the available keys in your ssh agent:

ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWtqI37jwxYMJ9XLq9VwHgJlhZViPVAGIUfMm8SAlfs6cka4Cj570lkoGK04r8JAVJFy/iKfhGpL9N9XuartfIoq6Cg/6Qvg3REupuqs51V2cBaC/gnWIQ7qZqlzBulvcOvzNfHFD/lX42J58+E8tWnYg6GzIsImFZQVpmI6SxNfSmVQIqxIufInrbQaI+pKXntdTQC9wyNK5FAA8TXAdff5ZDnmetsOTVble9Ia5m6gqM7MnxNZ56uDpn+6lCxRZSW+Ln2PDE7sivVcST4qpfwY4P4Lrb3vrjCGODFg4xmGNKXsLi2+uZbs5rW7bg4HFO50kKDucPV1M+rBWA9999 /usr/lib64/opensc-pkcs11.so

Enjoy!

Now you have a flexible yet robust way to authenticate your users which you can also extend by adding another type of authentication on your servers using PAM.

I recently worked a bit at how we could secure better our SSH connections to our servers at work.

So far we are using the OpenSSH public key only mechanism which means that there is no password set on the servers for our users. While this was satisfactory for a time we think that this still suffers some disadvantages such as:

  • we cannot enforce SSH private keys to have a passphrase on the user side
  • the security level of the whole system is based on the protection of the private key which means that it’s directly tied to the security level of the desktop of the users

This lead us to think about adding a 2nd factor authentication to SSH and about the usage of security keys.

Meet the Yubikey

Yubikeys are security keys made by Yubico. They can support multiple modes and work with the U2F open authentication standard which is why they got my attention.

I decided to try the Yubikey 4 because it can act as a smartcard while offering these interesting features:

  • Challenge-Response
  • OTP
  • GPG
  • PIV

Method 1 – SSH using pam_ssh + pam_yubico

The first method I found satisfactory was to combine pam_ssh authentication module along with pam_yubico as a 2nd factor. This allows server side passphrase enforcement on SSH and the usage of the security key to login.

TL;DR: two gotchas before we begin

ADVISE: keep a root SSH session to your servers while deploying/testing this so you can revert any change you make and avoid to lock yourself out of your servers.

Setup pam_ssh

Use pam_ssh on the servers to force usage of a passphrase on a private key. The idea behind pam_ssh is that the passphrase of your SSH key serves as your SSH password.

Generate your SSH key pair with a passphrase on your local machine.

ssh-keygen -f identity
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in identity.
Your public key has been saved in identity.pub.
The key fingerprint is:
SHA256:a2/HNCe28+bpMZ2dIf9bodnBwnmD7stO5sdBOV6teP8 alexys@yazd
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|                o|
|            . ++o|
|        S    BoOo|
|         .  B %+O|
|        o  + %+*=|
|       . .. @ .*+|
|         ....%B.E|
+----[SHA256]-----+

You then must copy your private key (named identity with no extension) to your servers under  the ~/.ssh/login-keys.d/ folder.

In your $HOME on the servers, you will get something like this:

.ssh/
├── known_hosts
└── login-keys.d
    └── identity

Then you can enable the pam_ssh authentication. Gentoo users should enable the pam_ssh USE flag for sys-auth/pambase and re-install.

Add this at the beginning of the file /etc/pam.d/ssh

auth    required    pam_ssh.so debug

The debug flag can be removed after you tested it correctly.

Disable public key authentication

Because it takes precedence over the PAM authentication mechanism, you have to disable OpenSSH PubkeyAuthentication authentication on /etc/ssh/sshd_config:

PubkeyAuthentication no

Enable PAM authentication on /etc/ssh/sshd_config

ChallengeResponseAuthentication no
PasswordAuthentication no
UsePAM yes

Test pam_ssh

Now you should be prompted for your SSH passphrase to login through SSH.

➜  ~ ssh cheetah
SSH passphrase:

Setup 2nd factor using pam_yubico

Now we will make use of our Yubikey security key to add a 2nd factor authentication to login through SSH on our servers.

Because the Yubikey is not physically plugged on the server, we cannot use an offline Challenge-Response mechanism, so we will have to use a third party to validate the challenge. Yubico gracefully provide an API for this and the pam_yubico module is meant to use it easily.

Preparing your account using your Yubikey (on your machine)

First of all, you need to get your Yubico API key ID from the following URL:

You will get a Client ID (this you will use) and Secret Key (this you will keep safe).

Then you will need to create an authorization mapping file which basically link your account to a Yubikey fingerprint (modhex). This is equivalent to saying “this Yubikey belongs to this user and can authenticate him”.

First, get your modhex:

Using this modhex, create your mapping file named authorized_yubikeys which will be copied to ~/.yubico/authorized_yubikeys on the servers (replace LOGIN_USERNAME with your actual account login name).

LOGIN_USERNAME:xxccccxxuuxx

NOTE: this mapping file can be a centralized one (in /etc for example) to handle all the users from a server. See the the authfile option on the doc.

Setting up OpenSSH (on your servers)

You must install pam_yubico on the servers. For Gentoo, it’s as simple as:

emerge sys-auth/pam_yubico

Copy your authentication mapping file to your home under the .yubico folder on all servers. You should get this:

.yubico/
└── authorized_yubikey

Configure pam to use pam_yubico. Add this after the pam_ssh on the file /etc/pam.d/ssh which should look like this now:

auth    required    pam_ssh.so
auth    required    pam_yubico.so id=YOUR_API_ID debug debug_file=/var/log/auth.log

The debug and debug_file flags can be removed after you tested it correctly.

Testing pam_yubico

Now you should be prompted for your SSH passphrase and then for your Yubikey OTP to login through SSH.

➜  ~ ssh cheetah
SSH passphrase: 
YubiKey for `ultrabug':

About the Yubico API dependency

Careful readers will notice that using pam_yubico introduces a strong dependency on the Yubico API availability. If the API becomes unreachable or your internet connection goes down then your servers would be unable to authenticate you!

The solution I found to this problem is to instruct pam to ignore the Yubikey authentication when pam_yubico is unable to contact the API.

In this case, the module will return a AUTHINFO_UNAVAIL code to PAM which we can act upon using the following syntax. The /etc/pam.d/ssh first lines should be changed to this:

auth    required    pam_ssh.so
auth    [success=done authinfo_unavail=ignore new_authtok_reqd=done default=die]    pam_yubico.so id=YOUR_API_ID debug debug_file=/var/log/auth.log

Now you can be sure to be able to use your Yubikey even if the API is down or unreachable 😉

April 30, 2017
Sebastian Pipping a.k.a. sping (homepage, bugs)

Hey there!

If you are not subscribed to the new Gentoo packages feed, let me quickly introduce you to SafeEyes that I started using on a daily basis. It has found it’s way into Gentoo as x11-misc/safeeyes now. This article does a good job:

SafeEyes Protects You From Eye Strain When Working On The Computer (webupd8.org)

Best, Sebastian

April 15, 2017
Gentoo Haskell Herd a.k.a. haskell (homepage, bugs)
GHC as a cross-compiler update (April 15, 2017, 11:05 UTC)

TL;DR:

Gentoo’s dev-lang/ghc-8.2.1_rc1 supports both cross-building and cross-compiling modes! It’s useful for cross-compiling haskell software and initial porting of GHC itself on a new gentoo target.

Building a GHC crossompiler on Gentoo

Getting ${CTARGET}-ghc (crosscompiler) on Gentoo:

# # convenience variables:
CTARGET=powerpc64-unknown-linux-gnu
#
# # Installing a target toolchain: gcc, glibc, binutils
crossdev ${CTARGET}
# # Installing ghc dependencies:
emerge-${CTARGET} -1 libffi ncurses gmp
#
# # adding 'ghc' symlink to cross-overlay:
ln -s path/to/haskell/overlay/dev-lang/ghc part/to/cross/overlay/cross-${CTARGET}/ghc
#
# # Building ghc crosscompiler:
emerge -1 cross-${CTARGET}/ghc
#
powerpc64-unknown-linux-gnu-ghc --info | grep Target
# ,("Target platform","powerpc64-unknown-linux")

Cross-building GHC on Gentoo

Cross-building ghc on ${CTARGET}:

# # convenience variables:
CTARGET=powerpc64-unknown-linux-gnu
#
# # Installing a target toolchain: gcc, glibc, binutils
crossdev ${CTARGET}
# # Installing ghc dependencies:
emerge-${CTARGET} -1 libffi ncurses gmp
#
# # Cross-building ghc crosscompiler:
emerge-${CTARGET} --buildpkg -1 dev-lang/ghc
#
# # Now built packages can be used on a target to install
# # built ghc as: emerge --usepkg -1 dev-lang/ghc

Building a GHC crossompiler (generic)

That’s how you get a powerpc64 crosscompiler in a fresh git checkout:

$ ./configure --target=powerpc64-unknown-linux-gnu
$ cat mk/build.mk
HADDOCK_DOCS=NO
BUILD_SPHINX_HTML=NO
BUILD_SPHINX_PDF=NO
# to speed things up
BUILD_PROF_LIBS=NO
$ make -j$(nproc)
$ inplace/bin/ghc-stage1 --info | grep Target
,("Target platform","powerpc64-unknown-linux")

Simple!

Below are details that have only historical (or backporting) value.

How did we get there?

Cross-compiling support in GHC is not a new thing. GHC wiki has a detailed section on how to build a crosscompiler. That works quite good. You can even target ghc at m68k: porting example.

What did not work so well is the attempt to install the result! In some places GHC build system tried to run ghc-pkg built for ${CBUILD}, in some places for ${CHOST}.

I never really tried to install a crosscompiler before. I think mostly because I was usually happy to make cross-compiler build at all: making GHC build for a rare target usually required a patch or two.

But one day I’ve decided to give full install a run. Original motivation was a bit unusual: I wanted to free space on my hard drive.

The build tree for GHC usually takes about 6-8GB. I had about 15 GHC source trees lying around. All in all it took about 10% of all space on my hard drive. Fixing make install would allow me to install only final result and get rid of all intermediate files.

I’ve decided to test make install code on Gentoo‘s dev-lang/ghc package as a proper package.

As a result a bunch of minor cleanups happened:

What works?

It allowed me to test various targets. Namely:

Target Bits Endianness Codegen
cross-aarch64-unknown-linux-gnu/ghc 64 LE LLVM
cross-alpha-unknown-linux-gnu/ghc 64 LE UNREG
cross-armv7a-unknown-linux-gnueabi/ghc 32 LE LLVM
cross-hppa-unknown-linux-gnu/ghc 32 BE UNREG
cross-m68k-unknown-linux-gnu/ghc 32 BE UNREG
cross-mips64-unknown-linux-gnu/ghc 32/64 BE UNREG
cross-powerpc64-unknown-linux-gnu/ghc 64 BE NCG
cross-powerpc64le-unknown-linux-gnu/ghc 64 LE NCG
cross-s390x-unknown-linux-gnu/ghc 64 BE UNREG
cross-sparc-unknown-linux-gnu/ghc 32 BE UNREG
cross-sparc64-unknown-linux-gnu/ghc 64 BE UNREG

I am running all of this on x86_64 (64-bit LE platform)

Quite a list! With help of qemu we can even test whether cross-compiler produces something that works:

$ cat hi.hs 
main = print "hello!"
$ powerpc64le-unknown-linux-gnu-ghc hi.hs -o hi.ppc64le
[1 of 1] Compiling Main             ( hi.hs, hi.o )
Linking hi.ppc64le ...
$ file hi.ppc64le 
hi.ppc64le: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, not stripped
$ qemu-ppc64le -L /usr/powerpc64le-unknown-linux-gnu/ ./hi.ppc64le 
"hello!"

Many qemu targets are slightly buggy and usually are very easy to fix!

A few recent examples:

  • epoll syscall is not wired properly on qemu-alpha: patch
  • CPU initialization code on qemu-s390x
  • thread creation fails on qemu-sparc32plus due to simple mmap() emulation bug
  • tcg on qemu-sparc64 crashes at runtime in static_code_gen_buffer()

Tweaking qemu is fun 🙂


April 10, 2017
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
py3status v3.5 (April 10, 2017, 10:19 UTC)

Howdy folks,

I’m obviously slacking a bit on my blog and I’m ashamed to say that it’s not the only place where I do. py3status is another of them and it wouldn’t be the project it is today without @tobes.

In fact, this new 3.5 release has witnessed his takeover on the top contributions on the project, so I want to extend a warm thank you and lots of congratulations on this my friend 🙂

Also, an amazing new contributor from the USA has come around in the nickname of @lasers. He has been doing a tremendous job on module normalization, code review and feedbacks. His high energy is amazing and more than welcome.

This release is mainly his, so thank you @lasers !

What’s new ?

Well the changelog has never been so large that I even don’t know where to start. I guess the most noticeable change is the gorgeous and brand new documentation of py3status on readthedocs !

Apart from the enhanced guides and sections, what’s amazing behind this new documentation is the level of automation efforts that @lasers and @tobes put into it. They even generate modules’ screenshots programmatically ! I would never have thought of it possible 😀

The other main efforts on this release is about modules normalization where @lasers put so much energy in taking advantage of the formatter features and bringing all the modules to a new level of standardization. This long work brought to light some lack of features or bugs which got corrected along the way.

Last but not least, the way py3status notifies you when modules fail to load/execute got changed. Now modules which fail to load or execute will not pop up a notification (i3 nagbar or dbus) but display directly in the bar where they belong. Users can left click to show the error and right click to discard them from their bar !

New modules

Once again, new and recurring contributors helped the project get better and offer a cool set of modules, thank you contributors !

  • air_quality module, to display the air quality of your place, by @beetleman and @lasers
  • getjson module to display fields from a json url, by @vicyap
  • keyboard_locks module to display keyboard locks states, by @lasers
  • systemd module to check the status of a systemd unit, by @adrianlzt
  • tor_rate module to display the incoming and outgoing data rates of a Tor daemon instance, by @fmorgner
  • xscreensaver module, by @lasers and @neutronst4r

Special mention to @maximbaz for his continuous efforts and help. And also a special community mention to @valdur55 for his responsiveness and help for other users on IRC !

What’s next ?

The 3.6 version will focus on the following ideas, some sane and some crazy 🙂

  • we will continue to work on the ability to add/remove/move modules in the bar at runtime
  • i3blocks and i3pystatus support, to embed their configurations and modules inside py3status
  • formatter optimizations
  • finish modules normalization
  • write more documentation and clean up the old ones

Stay tuned

March 25, 2017
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

We want to stabilize Perl 5.24 on Gentoo pretty soon (meaning in a few weeks), and do actually not expect any big surprises there. If you are running a stable installation, are willing to do some testing, and are familiar with our Gentoo bugzilla and with filing bug reports, then you might just be the right volunteer to give it a try in advance!

Here's what to do:

Step 1: Update app-admin/perl-cleaner to current ~arch.
I'm deliberately not supplying any version number here, since I might do another release, but you should at least have perl-cleaner-2.25.

Step 2: Make sure your system is uptodate (emerge -uDNav world) and do a depclean step (emerge --depclean --ask).

Step 3: Download the current stabilization list from bug 604602 and place it into your /etc/portage/package.keywords or /etc/portage/package.accept_keywords.

Step 4: Update your world (emerge -uDNav world), which triggers the perl update and the module rebuild.

Step 5: Run "perl-cleaner --all"  (you might also want to try "perl-cleaner --all --delete-leftovers").

... and make sure you file bugs for any problems you encounter, during the update and afterwards! Feedback is also appreciated if all goes fine; then you best leave a comment here on the blog post.

March 21, 2017
Jason A. Donenfeld a.k.a. zx2c4 (homepage, bugs)
WireGuard in Google Summer of Code (March 21, 2017, 18:52 UTC)

WireGuard is participating in Google Summer of Code 2017. If you're a student who would like to be funded this summer for writing interesting kernel code, studying cryptography, building networks, or working on a wide variety of interesting problems, then this might be appealing. The program opened to students on March 20th. If you're applying for WireGuard, choose "Linux Foundation" and state in your proposal that you'd like to work on WireGuard with "Jason Donenfeld" as your mentor.

WireGuard in Google Summer of Code (March 21, 2017, 18:52 UTC)

WireGuard is participating in Google Summer of Code 2017. If you're a student who would like to be funded this summer for writing interesting kernel code, studying cryptography, building networks, or working on a wide variety of interesting problems, then this might be appealing. The program opened to students on March 20th. If you're applying for WireGuard, choose "Linux Foundation" and state in your proposal that you'd like to work on WireGuard with "Jason Donenfeld" as your mentor.

March 17, 2017
Michał Górny a.k.a. mgorny (homepage, bugs)

You should know already that you are not supposed to rely on Portage internals in ebuilds — all variables, functions and helpers that are not defined by the PMS. You probably know that you are not supposed to touch various configuration files, vdb and other Portage files as well. What most people don’t seem to understand, you are not supposed to make any assumptions about the ebuild repository either. In this post, I will expand on this and try to explain why.

What PMS specifies, what you can rely on

I think the first confusing point is that PMS actually defines the repository format pretty thoroughly. However, it does not specify that you can rely on that format being visible from within ebuild environment. It just defines a few interfaces that you can reliably use, some of them in fact quite consistent with the repository layout.

You should really look as the PMS-defined repository format as an input specification. This is the format that the developers are supposed to use when writing ebuilds, and that all basic tools are supposed to support. However, it does not prevent the package managers from defining and using other package formats, as long as they provide the environment compliant with the PMS.

In fact, this is how binary packages are implemented in Gentoo. The PMS does not define any specific format for them. It only defines a few basic rules and facilities, and both Portage and Paludis implement their own binary package formats. The package managers expose APIs required by the PMS, and can use them to run the necessary pkg_* phases.

However, the problem is not limited to two currently used binary package formats. This is a generic goal of being able to define any new package format in the future, and make it work out of the box with existing ebuilds. Imagine just a few possibilities: more compact repository formats (i.e. not requiring hundreds of unpacked files), fetching only needed ebuild files…

Sadly, none of this can even start being implemented if developers continuosly insist to rely on specific repository layout.

The *DIR variables

Let’s get into the details and iterate over the few relevant variables here.

First of all, FILESDIR. This is the directory where ebuild support files are provided throughout src_* phases. However, there is no guarantee that this will be exactly the directory you created in the ebuild repository. The package manager just needs to provide the files in some directory, and this directory may not actually exist before the first src_* phase. This implies that the support files may not even exist at all when installing from a binary package, and may be created (copied, unpacked) later when doing a source build.

The next variable listed by the PMS is DISTDIR. While this variable is somewhat similar to the previous one, some developers are actually eager to make the opposite assumption. Once again, the package manager may provide the path to any directory that contains the downloaded files. This may be a ‘shadow’ directory containing only files for this package, or it can be any system downloads directory containing lots of other files. Once again, you can’t assume that DISTDIR will exist before src_*, and that it will exist at all (and contain necessary files) when the build is performed using a binary package.

The two remaining variables I would like to discuss are PORTDIR and ECLASSDIR. Those two are a cause of real mayhem: they are completely unsuited for a multi-repository layout modern package managers use and they enforce a particular source repository layout (they are not available outside src_* phases). They pretty much block any effort on improvement, and sadly their removal is continuously blocked by a few short-sighted developers. Nevertheless, work on removing them is in progress.

Environment saving

While we’re discussing those matters, a short note on environment saving is worth being written. By environment saving we usually mean the magic that causes the variables set in one phase function to be carried to a phase function following it, possibly over a disjoint sequence of actions (i.e. install followed by uninstall).

A common misunderstanding is to assume the Portage model of environment saving — i.e. basically dumping a whole ebuild environment including functions into a file. However, this is not sanctioned by the PMS. The rules require the package manager to save only variables, and only those that are not defined in global scope. If phase functions define functions, there is no guarantee that those functions will be preserved or restored. If phases redefine global variables, there is no guarantee that the redefinition will be preserved.

In fact, the specific wording used in the PMS allows a completely different implementation to be used. The package manager may just snapshot defined functions after processing the global scope, or even not snapshot them at all and instead re-read the ebuild (and re-inherit eclasses) every time the execution continues. In this case, any functions defined during phase function are lost.

Is there a future in this?

I hope this clears up all the misunderstandings on how to write ebuilds so that they will work reliably, both for source and binary builds. If those rules are followed, our users can finally start expecting some fun features to come. However, before that happens we need to fix the few existing violations — and for that to happen, we need a few developers to stop thinking only of their own convenience.

Marek Szuba a.k.a. marecki (homepage, bugs)
Gentoo Linux in a Docker container (March 17, 2017, 14:31 UTC)

I have been using Docker for ebuild development for quite a while and absolutely love it, mostly because how easy it is to manipulate filesystem state with it. Work on several separate ebuilds in parallel? Just spin up several containers. Clean up once I’m done? Happens automatically when I close the container. Come back to something later? One docker commit invocation and I’m done. I could of course do something similar with virtual machines (and indeed I have to for cross-platform work) – but for native amd64 is is extremely convenient.

There is, however, one catch. By default processes running in a Docker container are fairly restricted privilege-wise and the Gentoo sandbox uses ptrace(). Result? By default, certain ebuilds (sys-libs/glibc and dev-libs/gobject-introspection , to name just two) will fail to emerge. One can of course set FEATURES=”-sandbox -usersandbox” for such ebuilds but it is an absolute no-no for both new ebuilds and any stabilisation work.

In the past working around this issue required messing with Docker security policies, which at least I found rather awkward. Fortunately since version 1.13.0 there has been a considerably easier way – simply pass

--cap-add=SYS_PTRACE

to docker-run. Done! Sandbox can now use ptrace() to its heart’s content.

Big Fat Warning: The reason why by default Docker restricts CAP_SYS_PTRACE is that a malicious program can use ptrace() to break out of the container it runs in. Do not grant this capability to containers unless you know what you are doing. Seriously.

Unfortunately the above is not the end of the story because at least as of version 1.13.0, Docker does not allow to enhance the capabilities of a docker-build job. Why is this a problem? For my own work I use a custom image which extends somewhat the official gentoo/stage3-amd64-hardened . One of the things my Dockerfile does is rsync the Portage tree and update @world so that my image contains a fully up-to-date stage3 even when the official base image does not. You can guess what happens when Docker tries to emerge an ebuild requiring the sandbox to use ptrace()… and remember, one of the packages containing such ebuilds is sys-libs/glibc . To my current knowledge the only way around this is to spin up a ptrace-enabled container using the latest good intermediate image left behind by docker-build and execute the remaining build steps manually. Not fun… Hope they will fix this some day.