Gentoo Logo
Gentoo Logo Side
Gentoo Spaceship

. Aaron W. Swenson
. Agostino Sarubbo
. Alec Warner
. Alex Alexander
. Alex Legler
. Alexey Shvetsov
. Alexis Ballier
. Alexys Jacob
. Amadeusz Żołnowski
. Andreas K. Hüttel
. Andreas Proschofsky
. Anthony Basile
. Arun Raghavan
. Bernard Cafarelli
. Bjarke Istrup Pedersen
. Brent Baude
. Brian Harring
. Christian Ruppert
. Chí-Thanh Christopher Nguyễn
. Daniel Gryniewicz
. David Abbott
. Denis Dupeyron
. Detlev Casanova
. Diego E. Pettenò
. Domen Kožar
. Donnie Berkholz
. Doug Goldstein
. Eray Aslan
. Fabio Erculiani
. Gentoo Haskell Herd
. Gentoo Monthly Newsletter
. Gentoo News
. Gilles Dartiguelongue
. Greg KH
. Hanno Böck
. Hans de Graaff
. Ian Whyman
. Ioannis Aslanidis
. Jan Kundrát
. Jason Donenfeld
. Jeffrey Gardner
. Jeremy Olexa
. Joachim Bartosik
. Johannes Huber
. Jonathan Callen
. Jorge Manuel B. S. Vicetto
. Joseph Jezak
. Kenneth Prugh
. Kristian Fiskerstrand
. Lance Albertson
. Liam McLoughlin
. LinuxCrazy Podcasts
. Luca Barbato
. Luis Francisco Araujo
. Mark Loeser
. Markos Chandras
. Mart Raudsepp
. Matt Turner
. Matthew Marlowe
. Matthew Thode
. Matti Bickel
. Michael Palimaka
. Michal Hrusecky
. Michał Górny
. Mike Doty
. Mike Gilbert
. Mike Pagano
. Nathan Zachary
. Ned Ludd
. Nirbheek Chauhan
. Pacho Ramos
. Patrick Kursawe
. Patrick Lauer
. Patrick McLean
. Pavlos Ratis
. Paweł Hajdan, Jr.
. Peter Wilmott
. Petteri Räty
. Piotr Jaroszyński
. Rafael Goncalves Martins
. Raúl Porcel
. Remi Cardona
. Richard Freeman
. Robin Johnson
. Ryan Hill
. Sean Amoss
. Sebastian Pipping
. Steev Klimaszewski
. Stratos Psomadakis
. Sune Kloppenborg Jeppesen
. Sven Vermeulen
. Sven Wegener
. Thomas Kahle
. Tiziano Müller
. Tobias Heinlein
. Tobias Klausmann
. Tom Wijsman
. Tomáš Chvátal
. Vikraman Choudhury
. Vlastimil Babka
. Zack Medico

Last updated:
November 30, 2015, 13:04 UTC

Views expressed in the content published here do not necessarily represent the views of Gentoo Linux or the Gentoo Foundation.

Bugs? Comments? Suggestions? Contact us!

Powered by:
Planet Venus

Welcome to Gentoo Universe, an aggregation of weblog articles on all topics written by Gentoo developers. For a more refined aggregation of Gentoo-related topics only, you might be interested in Planet Gentoo.

November 29, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
lxc, ipv6 and iproute2 (November 29, 2015, 21:19 UTC)

Not so recently I got a soyoustart system since it is provided with an option to install Gentoo out of box.

The machine comes with a single ipv4 and a /64 amount of ipv6 addresses.


I want to use the box to host some of my flask applications (plaid mainly), keep some continuous integration instances for libav and some other experiments with compilers and libraries (such as musl, cparser other).

Since Diego was telling me about lxc I picked it. It is simple, requires not much effort and in Gentoo we have at least some documentation.

Setting up

I followed the documentation provided and it worked quite well up to a point. The btrfs integration works as explained, creating new Gentoo instances just worked, setting up the network… Required some effort.

Network woes

I have just 1 single ipv4 and some ipv6 so why not leveraging them? I decided to partition my /64 and use some, configured the bridge to take ::::1::1 and set up the container configuration like this: = veth = br0 = up = = auto = ::::1::4/80 = auto = 02:00:ee:cb:8a:04

But the route to my container wasn’t advertised.

Having no idea why I just kept poking around sysctl and iproute2 until I got:

  • sysctl.conf:
  net.ipv6.conf.all.forwarding = 1
  net.ipv6.conf.eth0.proxy_ndp = 1


ip -6 neigh add proxy ::::1::4 dev eth0

In my container runner script.

I know that at least other people had the problem so here this mini-post.

November 26, 2015
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)
Grafting history onto your Gentoo git clone (November 26, 2015, 23:14 UTC)

Somehow after a while I got a bit tired that my git checkout of the main Gentoo repository didn't have any real history available. So, here's how I got it back:

(Note, you may want a fast network connection for this.)

  • cd into the main directory of the Gentoo git checkout:
$ cd ~/Gentoo/gentoo
  • fetch 2GByte of converted cvs history into a new local branch "history-20150809-draft"
$ git fetch master:history-20150809-draft
  • attach the last commit of the cvs history to the first commit of the new git era
$ echo 56bd759df1d0c750a065b8c845e93d5dfa6b549d 2ebda5cd08db6bdf193adaa6de33239a83a73af0 > .git/info/grafts
And done. :)

Should at some point in the future a new, improved (or "official") conversion of the cvs history become available, here's (untested) what to do:
  • fetch it in the same way into a new local cvs history branch, and 
  • modify the grafts file to now connect the last commit of the new local cvs history branch with the first commit of the git era. 
Once you are happy with the result, you can delete the old local cvs history branch and run "git prune", freeing up the space used by the now obsolete old conversion.

Thanks to rich0 for providing the draft conversion (though inofficial so far) and to everyone else involved.

Alexys Jacob a.k.a. ultrabug (homepage, bugs)
py3status v2.7 (November 26, 2015, 18:52 UTC)

I’m more than two weeks late but I’m very glad to announce the release of py3status v2.7 which features a lot of interesting stuff !

For this release I want to salute the significant work and help of Daniel Foerster (@pydsigner), who discovered and fixed a bug in the event detection loop.

The result is a greatly improved click event detection and bar update speed with a largely reduced CPU consumption and less code !


  • major performance and click event detection improvements by Daniel Foerster
  • support of %z on time and tztime modules fixes #110 and #123 thx to @derekdreery and @olhotak
  • directive %Z and any other failure in parsing the time and tztime modules format will result in using i3status date output
  • add ethernet, wireless and battery _first_ instance detection and support. thx to @rekoil for reporting on IRC
  • i3status.conf parser handles configuration values with the = char

New modules

  • new rt module: display ongoing tickets from RT queues
  • new xsel module: display xsel buffers, by umbsublime
  • new window_title_async module, by Anon1234

Modules enhancements

  • battery_level module: major improvements, documentation, add format option, by Maxim Baz
  • keyboard_layout module: color customisation, add format option, by Ali Mousavi
  • mpd_status module: fix connection leak, by Thomas Sanchez
  • pomodoro module: implement format option and add additional display features, by Christoph Schober
  • spotify module: fix support for new versions, by Jimmy Garpehäll
  • spotify module: add support for colors output based on the playback status, by Sondre Lefsaker
  • sysdata module: trim spaces in `cpu_temp`, by beetleman
  • whatismyip module: change default check URL and make it configurable

Thanks !

Once again, thanks to all contributors listed above !

November 23, 2015
Hanno Böck a.k.a. hanno (homepage, bugs)

Delltl;dr Dell laptops come preinstalled with a root certificate and a corresponding private key. That completely compromises the security of encrypted HTTPS connections. I've provided an online check, affected users should delete the certificate.

It seems that Dell hasn't learned anything from the Superfish-scandal earlier this year: Laptops from the company come with a preinstalled root certificate that will be accepted by browsers. The private key is also installed on the system and has been published now. Therefore attackers can use Man in the Middle attacks against Dell users to show them manipulated HTTPS webpages or read their encrypted data.

The certificate, which is installed in the system's certificate store under the name "eDellRoot", gets installed by a software called Dell Foundation Services. This software is still available on Dell's webpage. According to the somewhat unclear description from Dell it is used to provide "foundational services facilitating customer serviceability, messaging and support functions".

The private key of this certificate is marked as non-exportable in the Windows certificate store. However this provides no real protection, there are Tools to export such non-exportable certificate keys. A user of the plattform Reddit has posted the Key there.

For users of the affected Laptops this is a severe security risk. Every attacker can use this root certificate to create valid certificates for arbitrary web pages. Even HTTP Public Key Pinning (HPKP) does not protect against such attacks, because browser vendors allow locally installed certificates to override the key pinning protection. This is a compromise in the implementation that allows the operation of so-called TLS interception proxies.

I was made aware of this issue a while ago by Kristof Mattei. We asked Dell for a statement three weeks ago and didn't get any answer.

It is currently unclear which purpose this certificate served. However it seems unliklely that it was placed there deliberately for surveillance purposes. In that case Dell wouldn't have installed the private key on the system.

Affected are only users that use browsers or other applications that use the system's certificate store. Among the common Windows browsers this affects the Internet Explorer, Edge and Chrome. Not affected are Firefox-users, Mozilla's browser has its own certificate store.

Users of Dell laptops can check if they are affected with an online check tool. Affected users should immediately remove the certificate in the Windows certificate manager. The certificate manager can be started by clicking "Start" and typing in "certmgr.msc". The "eDellRoot" certificate can be found under "Trusted Root Certificate Authorities". You also need to remove the file Dell.Foundation.Agent.Plugins.eDell.dll, Dell has now posted an instruction and a removal tool.

This incident is almost identical with the Superfish-incident. Earlier this year it became public that Lenovo had preinstalled a software called Superfish on its Laptops. Superfish intercepts HTTPS-connections to inject ads. It used a root certificate for that and the corresponding private key was part of the software. After that incident several other programs with the same vulnerability were identified, they all used a software module called Komodia. Similar vulnerabilities were found in other software products, for example in Privdog and in the ad blocker Adguard.

This article is mostly a translation of a German article I wrote for

Image source and license: Wistula / Wikimedia Commons, Creative Commons by 3.0

Update (2015-11-24): Second Dell root certificate DSDTestProvider

I just found out that there is a second root certificate installed with some Dell software that causes exactly the same issue. It is named DSDTestProvider and comes with a software called Dell System Detect. Unlike the Dell Foundations Services this one does not need a Dell computer to be installed, therefore it was trivial to extract the certificate and the private key. My online test now checks both certificates. This new certificate is not covered by Dell's removal instructions yet.

Dell has issued an official statement on their blog and in the comment section a user mentioned this DSDTestProvider certificate. After googling what DSD might be I quickly found it. There have been concerns about the security of Dell System Detect before, Malwarebytes has an article about it from April mentioning that it was vulnerable to a remote code execution vulnerability.

Update (2015-11-26): Service tag information disclosure

Another unrelated issue on Dell PCs was discovered in a tool called Dell Foundation Services. It allows webpages to read an unique service tag. There's also an online check.

November 21, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Code and Conduct (November 21, 2015, 18:54 UTC)

This is a sort of short list of checklists and few ramblings in the wake of Fosdem’s Code of Conduct discussions and the not exactly welcoming statements about how to perceive a Code of Conduct such as this one.

Code of Conduct and OpenSource projects

A Code of Conduct is generally considered a mean to get rid of problematic people (and thus avoid toxic situations). I prefer consider it a mean to welcome people and provide good guidelines to newcomers.

Communities without a code of conduct tend to reject the idea of having one, thinking that a is only needed to solve the above mentioned issue and adding more bureaucracy would just actually give more leeway to macchiavellian ploys.

That is usually a problem since, no matter how good things are now, it takes just few poisonous people to get in an unbearable situation and a you just need one in few selected cases.

If you consider the CoC a shackle or a stick to beat “bad guys” so you do not need it until you see a bad guy, that is naive and utterly wrong: you will end up writing something that excludes people due a quite understandable, but wrong, knee-jerk reaction.

A Code of Conduct should do exactly the opposite, it should embrace people and make easier joining and fit in. It should be the social equivalent of the developer handbook or the coding style guidelines.

As everybody can make a little effort and make sure to send code with spaces between operators everybody can make an effort and not use colorful language. Likewise as people would be more happy to contribute if the codebase they are hacking on is readable so they are more confident in joining the community if the environment is pleasant.

Making an useful Code of Conduct

The Code of Conduct should be a guideline for people that have no idea what the expected behavior, it should be written thinking on how to help people get along not on how to punish who do not like.

  • It should be short. It is pointless to enumerate ALL the possible way to make people uncomfortable, you are bound to miss it.
  • It should be understanding and inclusive. Always assume cultural bias and not ill will.
  • It should be enforced. It gets quite depressing when you have a 100+ lines code of conduct but then nobody cares about it and nobody really enforces it. And I’m not talking about having specifically designated people to enforce it. Your WHOLE community should agree on what is an acceptable behavior and act accordingly on breaches.

People joining the community should consider the Code of Conduct first as a request (and not a demand) to make an effort to get along with the others.


Since I saw quite some long and convoluted wall of text being suggested as THE CODE OF CONDUCT everybody MUST ABIDE TO, here some suggestion on what NOT do.

  • It should not be a political statement: this is a strong cultural bias that would make potential contributors just stay away. No matter how good and great you think your ideas are, those unrelated to a project that should gather people that enjoy writing code in their spare time should stay away. The Open Source is already an ideology, overloading it with more is just a recipe for a disaster.
  • Do not try to make a long list of definitions, you just dilute the content and give even more ammo to lawyer-type arguers.
  • Do not think much about making draconian punishments, this is a community on internet, even nowadays nobody really knows if you are actually a dog or not, you cannot really enforce anything if the other party really wants to be a pest.

Good examples

Some CoC I consider good are obviously the ones used in the communities I belong to, Gentoo and Libav, they are really short and to the point.


As I said before no matter how well written a code of conduct is, the only way to really make it useful is if the community as whole helps new (and not so new) people to get along.

The rule of thumb “if somebody feels uncomfortable in a non-technical discussion, once he says, drop it immediately”, is ok as long:
* The person uncomfortable speaks up. If you are shy you might ask somebody else to speak up for you, but do not be quiet when it happens and then fill a complaint much later, that is NOT OK.
* The rule is not bent to derail technical discussions. See my post about reviews to at least avoid this pitfall.
* People agree to drop at least some of their cultural biases, otherwise it would end up like walking on eggshells every moment.

Letting situations going unchecked is probably the main issue, newcomers can think it is OK to behave in a certain way if people are behaving such way and nobody stops that, again, not just specific enforcers of some kind, everybody should behave and tell clearly to those not behaving that they are problematic.

Gentoo is a big community so once somebody steps the boundaries gets problematic having a swift reaction, lots of people prefer not to speak up when something happens, so people unwillingly causing the problem are not made aware immediately.

The people then in charge to dish bans have to try to figure out what exactly was wrong and there the cultural biases everybody has might or might not trigger and make the problem harder to address.

Libav is a much smaller community and in general nobody has qualms in saying “please stop” (that is also partially due how the community evolved).

Hopefully this post would help avoid making some mistakes and help people getting along better.

November 15, 2015
Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
Memories of the future of the past (November 15, 2015, 13:32 UTC)

Every now and then I lament the badness of current things, and then I remember the things we had that we can't get anymore ...

  • ISDN Telephones were a wonderful upgrade to analog Telephones. 8kB/s dedicated bandwidth with low latency. Compared to that all VoIP things I've used were just horribly cheap shoddy crap of unspeakably bad quality. Luckily ISDN has been discontinued and is no longer available to consumers, so we have no reference for what good audio quality means.
  • High-res displays like the IBM T220. Look it up, it's a time traveller! And, of course, it was discontinued, with no modern device coming close.
  • Mobile phones that we recharged every week like the Motorola StarTac I bought a few years ago. Now 24h seems to be 'ok' ...
  • Washing machines that took 30 minutes for one load which is not energy-efficient, so instead the modern ones run for 1-2h. Not sure how that helps, and it looks like they use more water too. So we just hide the problem and PROBLEM SOLVED?
  • ThinkPad Notebooks
And many other things that were better in the past, but have regressed now to a lower quality, less-feature, harder to repair state.

Can we please have more future?

November 14, 2015
Employment in a technological era (November 14, 2015, 14:35 UTC)

Lately I've been spending some time reading up on research into developments to the nature of employment given the increased computerization and automation in today's, and in particular, in tomorrow's world. These developments brings immense increases in productivity and opens up a new world of opportunities, but are employees keeping track and updating their skill sets to utilize it? My personal opinion is no, which was what initiated looking into the research on the matter.

Frey and Osborne's paper "The future of employment: how suspectible are jobs to computerisation" (2013) bring some interesting aspects, including a decent historical context to this issue; starting with referencing how John Maynard Keynes is frequently cited for his prediction of a widespread technological unemployment "due to our discovery of means of economising the use of labor outrunning the pace of which we can find new uses for labor" (Keynes, 1933). This was of course during a different technological advancement than we're experiencing now,  but it shows that the discussion is not new, in fact it is nicely illustrated by an example of William Lee, inventing the stocking frame knitting machine in 1589, hoping that it would relieve workers of hand-knitting, something which met opposition by Queen Elizabeth I that was more concerned with the employment impact and refused to grant him a patent, claiming that "Thou aimest high, Master Lee. Consider thou what the invention could do to my poor subjects. It would assuredly bring to them ruin by depriving them of employment, thus making them beggars" (cited in Acemoglu and Robinson, 2012).

Has anything changed since the 16th century, or are we facing the same kind of social opposition to changing the status quo? How many, today, are willing to learn a programming language in order to interface with and utilize the tools of today? As pointed out by Makyr (1998): "Unless all individuals accept the "verdict" of the market outcome, the decision whether to adopt an innovation is likely to be resisted by losers through non-market mechanisms and political activism". This was followed up by the luddite riots between 1811 and 1816 as a manifestation of a fear of technological change among workers as Parliament revoked a 1551 law prohibiting the use of gig mills in the wool-finishing trade.

Today's challenges to labor markets are different in form, yet resemble the historical aspects to a great extent. These days the ability to communicate with a computer is, in my humble opinion, as vital as learning human languages, yet there are barely a few pushes towards learning programming languages alongside human spoken languages. My hypothesis is that a reason for this is a lack of knowledge in the adult population for the same, and quite frankly mathematics and logic in general, which naturally makes people uncomfortable requiring children to learn these subjects. Initiatives such as the UK's attempt to get kids coding, with changes to the national curriculum. ICT – Information and Communications Technology introducing a new “computing” curriculum including coding lessons for children as young as five (September 2013) is therefore very welcome, but as referenced in an article in The Guardian: "it seems many parents will be surprised when their children come home from school talking about algorithms, debugging and Boolean logic" and "It's about giving the next generation a chance to shape their world, not just be consumers in it".

The ability to shape my own day is one of the reasons why I'm personally interested in the world of open source. If I'm experiencing an issue while running an application, or if I want to extend it with new functionality, it is possible to do something about it when the source is available. Even more so, in a world that is increasingly complex and interconnected, basing this communication on open standards enables participation from multiple participants across different operating systems and user interfaces.

At the same time, increasingly so in the aftermath of Edward Snowden, I want to have the ability to see what happens with my data. Reading through the End User License Agreements (EULA) of services being offered to consumers I sometimes get truly scared. The last explicit example was the music playing service Spotify that introduced new terms stating that in order to continue using the service I would have to accept to having gained permission from all contacts to share their personal information. Safe to say I terminated that subscription.

There is an increasing gap in the knowledge required to understand the ramifications of the services being developed, the value of private information, and people's ability to recognize what is happening in an ever connected world. As pointed out in two earlier posts, "Your Weakest Security Link? Your Children, or is it?" and "Some worries about mobile appliances and the Internet of Things" this can actually be quite difficult, with the end result of individuals just drifting along.

So what do you think? Why not pick up an online tutorial on learning SQL, the structured query language used to talk with most database systems the next time you're feeling bored and is inclined to put on a TV program or just lie back on the couch, or maybe pick up a little bit of python, C, or for that matter C# if you're in a windows-centric world. Or as a general plea; make read a book once in a while.


November 13, 2015
Michał Górny a.k.a. mgorny (homepage, bugs)
The Ultimate Guide to EAPI 6 (November 13, 2015, 21:13 UTC)

Now that EAPI 6 is Council-approved and pretty close to being deployed, I think it’s about time to write up a not-so-short guide to it. It’s especially important that EAPI 6 is a bit different from the previous EAPIs. It was not only seen as an opportunity to add new features but also to tidy things up a bit and improve strictness.

If you look into the PMS, you’d see that we’d not only added completely new stuff but also ported a few common eclass functions, in a little cleaner form, and new useful phase functions. Furthermore, we used EAPI 6 as an opportunity to finally increase strictness of Portage in regard to PMS, and ban some long-unwanted eclasses.

Therefore, I’d like to ask you — please don’t go hooray-enabling EAPI 6 support in your ebuilds and eclasses. Take a while to think, and do things right. Please think of EAPI 6 as a lifetime opportunity of improving your eclass APIs, as we improved the API provided by Package Managers.

Now, time for a little summary of changes, their implications and a bit of rationale.

Bash upgrade and compatibility setting

The first important change in EAPI 6 is the upgrade to bash-4.2 (the earlier EAPIs required bash-3.2). At a first glance, it just brings a few nice quirks like cleaner fd magic (used by multiprocessing.eclass), case conversion for variables and associative arrays. However, aside to that bash compatibility setting is forced — that could hopefully prevent existing code from breaking in future bash versions.

What does this solve, you ask? For example, it solves the tilde substitution issue:


The above snippet gives the expected result only when executed with bash-4.2 and lower. However, starting with bash-4.3 it performs tilde expansion on right-hand argument of the pattern substitution and puts your home directory in there. You can disable that via backslash-escaping the argument — but then it puts an extra backslash in older bash versions! How frustrating…

Now, if we have BASH_COMPAT fixed by the PM at 4.2, this discrepancy is no longer an issue. Because all supported versions of bash will force the old behavior, you can write the code without any conditionals, workarounds or worrying that it will suddenly break sometime in the future.

failglob in global scope

The second change to ebuild syntax is that filename expansion is now an explicit error in global scope. While this may sound scary at first, it wasn’t ever really supported. Note that this doesn’t affect phase functions, just the code executed when sourcing the ebuild. This is mostly targeted towards snippets like the following:

inherit python-r1

RDEPEND="$(python_gen_cond_dep 'dev-python/foo[${PYTHON_USEDEP}]' python2*)"

Do you see the problem here? python2* lacks quoting. This usually doesn’t cause any issues but if you happened to be in a directory with pythonFOOBAR file while sourcing the ebuild, it would suddenly pick it up and return pretty unexpected results. As a solution, an EAPI=6 ebuild would instantly abort here and require you to quote the pattern.

LC_COLLATE and LC_CTYPE settings

The description for locale settings may seem a little blurry but that’s for a reason. The original idea was that the Package Manager would export (pretty much like Paludis does):


to enforce stable sorting and case conversions. Without the former, some locales could cause results of globs such as patch files to apply in different order. Without the latter, some locales could give unexpected results of case changes. In particular, the Turkish tr_TR.UTF-8 locale is only too happy to map lowercase ‘i’ to uppercase ‘İ’ (yep, it has a dot above!)

So why is it worded this muddy? Because of our lovely Python and the lot of high quality scripts that are only too happy to interpret read files in the charset corresponding to the current locale, and therefore bail out hard when they don’t conform to the ASCII charset implied by the C locale.

So instead of C, we need a C.UTF-8 which hasn’t made its way into glibc for… how long exactly? Not to mention the portability. Therefore, we just request a seemingly sane locale, and hope we can work something out. In implementation, the Package Manager will most likely set LC_COLLATE=C directly and complain to the user if his LC_CTYPE does not conform.

Nevertheless, in EAPI 6 you should be able to safely remove your local LC_COLLATE and LC_CTYPE redefinitions, and hopefully also most of those excessive LC_ALL settings.

eapply, eapply_user and src_prepare()

This is probably the most controversial feature of the new EAPI. Those functions are intended as reasonably simple and predictable replacements for the commonly used eutils.eclass beasts, with new default src_prepare() to finish off the long-deprecated base.eclass.

eapply has a few substantial differences from epatch. However, I hope that people will nevertheless decide to abandon the latter, and use the new function, effectively also reducing the use of eutils.eclass. The differences are:

  • No -p* guessing. Instead, it defaults to -p1 and lets you override it. Now patches can apply predictably.
  • No magical configuration variables. All options are passed as parameters, followed by files. If options need to take separate arguments, -- is provided to separate options from files.
  • No compression support. There’s unpack that usually does that for you, you know, and patches in ${FILESDIR} are never compressed.
  • Simple directory handling. It just applies all *.diff and *.patch files from a directory. No custom suffixes, no exclusions.

While this may sound like a lot was lost, the function is still quite big and even twice as useful. Most importantly, all patches now apply (or fail) pretty predictably, and if they do fail, you get a clear output.

eapply_user provides user patch support on top of eapply. However, since user patches are a matter of configuration, the PMS really leaves the details to the Package Manager, and doesn’t even prohibit it from extending patching beyond eapply. eapply_user is considered obligatory, and can be safely called more than once (applying patches only the first time) to make eclass writing easier.

Both function are used in the default src_prepare() implementation which was inspired by the common implementations shared between multiple eclasses. In particular, it applies patches from the PATCHES variable (array strongly preferred due to ${FILESDIR}) and then user patches. Please note that in some cases, it would be advised to remove src_prepare() from your eclass altogether rather than calling the default from it.

einstalldocs and src_install()

This one’s much simpler and accepted in full agreement. It combines two categories of requests created against the documentation install code introduced in EAPI 4. Firstly, it splits the code off default_src_install() so that it can be reused without implicitly calling emake install. Secondly, it solves a number of problems with the original implementation.

So what’s the story here? EAPI 4 introduced a very nice src_install() phase which was able to install a few default documentation pieces automatically. That was quite liked, except that it was tightly bound with the default emake install call. As a result, a number of eclasses ended up (still) redefining it, with the most noticeable example of base.eclass having a split phase function for it. But since the eclass was long deprecated already, other eclasses had to copy the code rather than reuse it.

The idea of having a split function arose early during EAPI 6 work. While writing a new function, the issues and requests against the current default were collected and considered. Finally, the new function was given to early Council approval, and committed to eutils.eclass as a backport of the future EAPI 6 function. And as you can see now, it lands in EAPI 6 in exactly the same form, replacing the eutils implementation.

As a reminder, the changes from the original EAPI 4 implementation are:

  • empty DOCS variable disables installing documentation completely. Previously, it was undefined behavior (usually a dodoc usage error).
  • The DOCS variable can name directories as well as files.
  • HTML_DOCS variable has been added. It installs the listed files and/or directories (using dodoc, not deprecated dohtml!) into the html/ subdirectory of docdir.

As a result, the default src_install() implementation differs by the documentation install changes listed above. If you were overriding this phase just to get einstalldocs into it, you can remove it. If you were using einstalldocs from eutils.eclass, you do not have to inherit it anymore. And if you were inlining your own documentation install code, it is a good idea to replace it with standard one in EAPI 6.

get_libdir and econf changes

The next major target for EAPI 6 changes was the econf function, commonly used in src_configure(). Here, two goals were achieved. Firstly, a get_libdir function inspired by multilib.eclass was added. Secondly, --docdir and --htmldir parameters are now passed to configure scripts.

Why would get_libdir be related to econf at all? Well, except for the fact that it’s often used in configure script arguments, it pretty much shares the logic used to figure out libdir. So we’re not just moving a commonly used eclass function to the EAPI — we’re splitting the libdir determination logic off econf, and sharing it with ebuilds.

As for the two new parameters — --docdir and --htmldir — I don’t think they really need any comment here.


This one’s very controversial as well, and I’d personally prefer not having to introduce it at all. However, without it the developers would still resort to ugly hacks, so better to have a safe alternative. So, what’s the background?

Some developers found it quite convenient to do some conditionals in eclasses based on what ends up in IUSE. For example, if the ebuild has IUSE=static-libs, we magically enable code controlling static library build and install. Except for being ugly, this design had two major issues. Firstly, the code checking that was inconsistent and often buggy — some eclasses handled +static-libs, some didn’t. Secondly, the specification never mandated IUSE having any sane value throughout phase functions. So the results could have been quite surprising.

EAPI 6 solves both of those issues by providing a standard in_iuse function. All uses implementation-defined magic to check whether the specified flag is listed in the IUSE_EFFECTIVE conceptual variable. This includes not only flags set by ebuilds and eclasses but also implicitly added by profiles — in other words, any flag that is legal to the use function call.

I should note that in_iuse is legal only in phase functions — it is disallowed in global scope (i.e. while IUSE is still subject to changes). Nevertheless, I would heartily recommend avoiding it and using explicit conditional variables instead.

nonfatal die

Did you happen to dislike how some eclass-defined helpers could not be nonfatal properly? That’s no longer the case with EAPI 6. The die function has gained a simple -n option which makes it respect nonfatal, and therefore be a no-op when it is in effect.

In other words, you can write helpers like the following:

efoo() {
	foo || die -n "foo failed" || return ${?}

where die would simply return false and continue execution if nonfatal is in effect. Please remember that ‘nonfatal die’ does not return from the function implicitly. If you need to do so, you need to explicitly call return like in the above snippet.

unpack changes

Now we’re past the most interesting changes. For completeness, there were few small requests accepted for the unpack function:

  • Absolute and non-prefixed relative paths can be used now. The path handling is still a little creepy since pure filename (i.e. without a single slash) implicitly references ${DISTDIR} but anything with a slash is treated as a regular path.
  • .txz suffix is interpreted equivalently to .tar.xz. Previously, only the long form was accepted.
  • Suffix matching is case-insensitive now. Which means that .ZIP works as well as .zip. Hooray, we can now handle 8.3 filenames cleanly!

einstall banned, dohtml deprecated

Aside to new features, there are two important removals to be noted. The long-discouraged einstall function has been finally banned, and the dohtml function has been deprecated with the goal of removing it in EAPI 7.

The einstall command was intended as a cheap work-around for build systems that did not support DESTDIR= argument to emake install. Instead, it redefined a number of direct install paths (such as bindir=), passing the staging area path and target directory combined. Sadly, it has often been cause of confusion for new developers who have seen it as the standard install command (in line with econf, emake…).

Over a year ago, I’ve started tracking uses of einstall on behalf of QA, and noticed that most of the uses were completely unnecessary, either because the build system in question handled DESTDIR= correctly, or had a custom variable serving the same purpose. Furthermore, the function used parameter names matching autoconf while it was intended to deal with custom build systems which may require other variables. Therefore, it was decided that it is better to ban the command and inline the relevant make parameters whenever really unavoidable.

The dohtml function has been the official way of installing HTML documentation for quite a long time. No other command defined by PMS had as many options. Confusingly to some developers, it provided implicit filtering by file suffixes, with additional options to add additional suffixes, replace suffix list, filter by filename, exclude directories and directly alter install path. Moreover, there was yet another enhancement request asking for new suffixes in the default filter.

The Council has agreed with us that the function is overcomplex and confusing, and will be an object of never-ending requests and updates. Most of Gentoo developers have seen it as a way of installing files in the html/ subdirectory of docdir while it did so much more — and potentially confused them by excluding some files. It also resulted in absurd situations like people moving gtk-doc files out of the standard location used by gtk-doc just to match the ‘standard’ established by the function.

dohtml is officially deprecated since EAPI 6, and is expected to be banned in EAPI 7. The suggested replacement is dodoc which is much simpler, faster and in many cases will work the same (or even better, by restoring the files that were mistakenly missed by dohtml). It is possible that the current dohtml will eventually be moved to an eclass for the ebuilds that require filtering, yet the fate of this idea is unclear yet.

To help with the migration, Portage provides the following quirk (for make.conf or environment):


When run with this, each dohtml invocation will verbosely list all files that were skipped through filtering. A null output indicates that dohtml can be safely replaced by dodoc.

Global-scope helpers banned in Portage

EAPI 6 brings not only the spec changes but also new strictness to Portage. In particular, almost all helper functions are now banned in global scope, as they are banned by PMS and other package managers. What does this imply? Most importantly, toolchain’s use of USE=multislot is now officially banned and will no longer work in EAPI 6.

Eclasses banned in EAPI 6

Finally, the changes brought by EAPI 6 will likely result in a number of API changes in multiple eclasses. Moreover, some of the existing eclasses will become completely unnecessary and will be banned.

The most important example here is the base.eclass which was semi-officially deprecated for a long time. In EAPI 6, it is fully replaced by default phase functions and must not be used. Similarly, I’ve decided to discontinue and ban autotools-utils.eclass and autotools-multilib.eclass, the former serving a similar purpose as base.eclass did in the past and the latter being just a wrapper atop multilib-minimal.eclass.

Aside to that, I’ve banned a number of eclasses that are deprecated currently, to avoid their use being carried on to the next EAPI.


To summarize all the changes in EAPI 6, let me assemble a check list of tasks that need to be done when updating ebuilds to EAPI 6:

  1. You can update your bash code to use bash-4.2 features, and remove unnecessary conditionals for future-incompatible behavior. It is unlikely that you have any.
  2. Ensure that all patterns that are supposed to be passed as-is to functions are quoted.
  3. Remove unnecessary LC_COLLATE and LC_CTYPE if they only reset it to the C (or POSIX) locale.
  4. If you are using epatch, try hard to replace it with eapply. You may need to pass an explicit -p value.
  5. If you are using epatch_user, you have to replace it with eapply_user. If you are not and you define src_prepare(), you have to add it there.
  6. If you are left with src_prepare() that looks pretty much like the default one, drop it. It is also a good idea to remove unnecessary src_prepare() definitions from eclasses.
  7. If you are using einstalldocs, this function is now part of EAPI. Don’t count it as a reason to inherit eutils.eclass. If you are using a similar inline code, it’s about time you switch to einstalldocs.
  8. If you are using in_iuse, this function is now part of EAPI and even really legal. Don’t count it as a reason to inherit eutils.eclass. In fact, hurry to switch to EAPI 6 to get a predictable behavior!
  9. Check if you are still using anything from eutils.eclass. If you are not, remove the inherit and save a few cycles.
  10. If you are using get_libdir, this function is now part of EAPI. Don’t count it as a reason to inherit multilib.eclass. If you are not using anything else from multilib.eclass, remove the inherit.
  11. If you are passing standard --docdir and/or --htmldir to econf, you can remove them. EAPI 6 does that for you.
  12. If you are defining ebuild helpers that should respect nonfatal, you should update appropriate die to use die -n || return. However, please note that it’s usually a good idea to use regular die on usage errors, and respect nonfatal only for external errors.
  13. If you are using hacks to support .txz, uppercase archive suffixes like .ZIP or extracting files by absolute or relative paths, you can remove them and use new unpack features.
  14. If you are using einstall, remove it. I’m pretty sure you never really needed it. And if you really did, then you most likely missed some custom prefix option in upstream build system.
  15. If you are using dohtml, consider replacing it with dodoc -r. You can set PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES=yes before doing that, to see if the two will be equivalent.
  16. If you are inheriting base.eclass, autotools-utils.eclass or any of the other banned eclasses, make sure to remove it from your eclasses and ebuilds.

Sebastian Pipping a.k.a. sping (homepage, bugs)
Fwd: The Butterfly Effect (band) (November 13, 2015, 04:49 UTC)


Among the top rock albums I have run into so far are

by a band called “The Butterfly effect“. I consider them a bit lesser known, but I don’t have data to back it up. One of my long time favourites. If you can use a hint, check them out.

For the embedded one, start with track #2 at 7:04.

November 09, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)
Babies or tiny drunks? (November 09, 2015, 23:14 UTC)

Recently I found this great thread on Reddit about two toddlers being like drunk girls. Though the animated GIF was entertaining, one particular comment in the thread was pure gold:

Babies are born blackout drunk and slowly sober as they age.

They form no permanent memories for several years, have no hand-eye coordination, and are either crying or sleeping. Slowly they regain the ability to crawl around on all fours, then to stumble drunkenly across the room before falling on their faces.

Eventually they become sober enough to start speaking slurred, broken sentences. But their emotions are still totally out of control, and they’ll fly into a rage, a sobbing fit, or uncontrollable giggles at the slightest provocation.

Puberty is the hangover.

You can see the comment and context here.

Too funny… and accurate.


November 08, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Trusting the context (November 08, 2015, 21:57 UTC)

This mini-post spurred from this bug.

AVFrame and AVCodecContext

In Libav there are a number of patterns shared across most of the components.
Does not matter if it models a codec, a demuxer or a resampler: You interact with it using a Context and you get data in or out of the module using some kind of Abstraction that wraps data and useful information such as the timestamp. Today’s post is about AVFrames and AVCodecContext.


The most used abstraction in Libav by far is the AVFrame. It wraps some kind of raw data that can be produced by decoders and fed to encoders, passed through filters, scalers and resamplers.

It is quite flexible and contains the data and all the information to understand it e.g.:

  • format: Used to describe either the pixel format for video and the sample format for audio.
  • width and height: The dimension of a video frame.
  • channel_layout, nb_samples and sample_rate for audio frames.


This context contains all the information useful to describe a codec and to configure an encoder or a decoder (the generic, common features, there are private options for specific features).

Being shared with encoder, decoder and (until Anton’s plan to avoid it is deployed) container streams this context is fairly large and a good deal of its fields are a little confusing since they seem to replicate what is present in the AVFrame or because they aren’t marked as write-only since they might be read in few situation.

In the bug mentioned channel_layout was the confusing one but also width and height caused problems to people thinking the value of those fields in the AVCodecContext would represent what is in the AVFrame (then you’d wonder why you should have them in two different places…).

As a rule of thumb everything that is set in a context is either the starting configuration and bound to change in the future.

Video decoders can reconfigure themselves and output video frames with completely different geometries, audio decoders can report a completely different number of channels or variations in their layout and so on.

Some encoders are able to reconfigure on the fly as well, but usually with more strict constraints.

Why their information is not the same

The fields in the AVCodecContext are used internally and updated as needed by the decoder. The decoder can be multithreaded so the AVFrame you are getting from one of the avcodec_decode_something() functions is not the last frame decoded.

Do not expect any of the fields with names similar to the ones provided by AVFrame to stay immutable or to match the values provided by the AVFrame.

Common pitfalls

Allocating video surfaces

Some quite common mistake is to use the AVCodecContext coded_width and coded_height to allocate the surfaces to present the decoded frames.

As said the frame geometry can change mid-stream, so if you do that best case you have some lovely green surrounding your picture, worst case you have a bad crash.

I suggest to always check that the AVFrame dimensions fit and be ready to reconfigure your video out when that happens.

Resampling audio

If you are using a current version of Libav you have avresample_convert_frame() doing most of the work for you, if you are not you need to check that format channel_layout and sample_rate do not change and manually reconfigure.

Rescaling video

Similarly you can misconfigure swscale and you should check manually that format, width and height and reconfigure as well. The AVScale draft API on provides an avscale_process_frame().

In closing

Be extra careful, think twice and beware of the examples you might find on internet, they might work until they wont.

November 07, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
New cvechecker release (November 07, 2015, 10:07 UTC)

A short while ago I got the notification that pulling new CVE information was no longer possible. The reason was that the NVD site did not support uncompressed downloads anymore. The fix for cvechecker was simple, and it also gave me a reason to push out a new release (after two years) which also includes various updates by Christopher Warner.

So cvechecker 3.6 is now available for general consumption.

November 06, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Reviews (November 06, 2015, 17:03 UTC)

This spurred from some events happening in Gentoo, since with the move to git we eventually have more reviews and obviously comments over patches can be acceptable (and accepted) depending on a number of factors.

This short post is about communicating effectively.

When reviewing patches

No point in pepper coating

Do not disparage code or, even worse, people. There is no point in being insulting, you add noise to the signal:

You are a moron! This is shit has no place here, do not do again something this stupid.

This is not OK: most people will focus on the insult and the technical argument will be totally lost.

Keep in mind that you want people doing stuff for the project not run away crying.

No point in sugar coating

Do not downplay stupid mistakes that would crash your application (or wipe an operating system) because you think it would hurt the feelings of the contributor.

    rm -fR /usr /local/foo

Is as silly as you like but the impact is HUGE.

This is a tiny mistake, you should not do that again.

No, it isn’t tiny it is quite a problem.

Mistakes happen, the review is there to avoid them hitting people, but a modicum of care is needed:
wasting other people’s time is still bad.

Point the mistake directly by quoting the line

And use at most 2-3 lines to explain why it is a problem.
If you can’t better if you fix that part yourself or move the discussion on a more direct media e.g. IRC.

Be specific

This kind of change is not portable, obscures the code and does not fix the overflow issue at hand:
The expression as whole could still overflow.

Hopefully even the most busy person juggling over 5 different tasks will get it.

Be direct

Do not suggest the use of those non-portable functions again anyway.

No room for interpretation, do not do that.

Avoid clashes

If you and another reviewer disagree, move the discussion on another media, there is NO point in spamming
the review system with countless comments.

When receiving reviews (or waiting for them)

Everybody makes mistakes

YOU included, if the reviewer (or more than one) tells you that your changes are not right, there are good odds you are wrong.

Conversely, the reviewer can make mistakes. Usually is better to move away from the review system and discuss over emails or IRC.

Be nice

There is no point in being confrontational. If you think the reviewer is making a mistake, politely point it out.

If the reviewer is not nice, do not use the same tone to fit in. Even more if you do not like that kind of tone to begin with.

Wait before answering

Do not update your patch or write a reply as soon as you get a notification of a review, more changes might be needed and maybe other reviewers have additional opinions.

Be patient

If a patch is unanswered, ping it maybe once a week, possibly rebasing it if the world changed meanwhile.

Keep in mind that most of your interaction is with other people volunteering their free time and not getting anything out of it as well, sometimes the real-life takes priority =)

November 05, 2015
Mike Pagano a.k.a. mpagano (homepage, bugs)
kdbus not in gentoo-sources (November 05, 2015, 23:33 UTC)

As per upstream developers instructions to Fedora maintainers [1] , kdbus will not be included in any new gentoo-sources genpatches release.

Of course, I will revisit when upstream feels it’s ready to be tested in distributions once again.


November 02, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Tags in git (November 02, 2015, 11:46 UTC)

mini-post about using tags in git commits.


In git a commit message is structured in a first subject line, an empty newline and more text making the body of the message.

The subject can be split in two components tags and the actual subject.

tag1: tag2: Commit Subject

A body with more information spanning
multiple lines.

The tags can be used to pin the general area the patch is impacting, e.g:

ui: Change widget foo


When you are looking at the history using git log having tags helps a lot digging out old commits, for example: you remember some commit added some timeout system in something related to the component foo.

git log --oneline | grep foo:

Would help figuring out the commit.

This usage is the best when working with not well structured codebase, since alternatively you can do

git log --oneline module/component

If you use separate directories for each module and component within the module.

PS: This is one of the reasons plaid focuses a lot on tags and I complain a lot when tags are not used.

October 30, 2015
Arun Raghavan a.k.a. ford_prefect (homepage, bugs)
PulseAudio 7.1 is out (October 30, 2015, 13:31 UTC)

We just rolled out a minor bugfix release. Quick changelog:

  • Fix a crasher when using srbchannel
  • Fix a build system typo that caused symlinks to turn up in /
  • Make Xonar cards work better
  • Other minor bug fixes and improvements

More details on the mailing list.

Thanks to everyone who contributed with bug reports and testing. What isn’t generally visible is that a lot of this happens behind the scenes downstream on distribution bug trackers, IRC, and so forth.

October 29, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)
Happy 18th Birthday, Noah (October 29, 2015, 05:01 UTC)

Happy 18th Birthday, Noah! I hope that you have a great day today, and that this upcoming year is your best one yet! Not too long ago, I saw a video of someone solving a Rubik’s cube blindfolded, (after only looking at the cube for about a minute), and it made me think back to solving them together. It also made me think of the cube that we made using photos, and not realising that it would be absurdly more difficult than ones with just colours—since you would have to have each square oriented correctly as well.

Noah's custom Rubik's cube for his birthday in 2009
Click to enlarge

Remember that?

Anyway, I hope that you have a great birthday! Maybe we’ll get to talk sometime soon. I would like to hear about all the things that have gone on over the last six years, and especially what your plans are for college.

Love ya’, kid,

October 28, 2015
Arun Raghavan a.k.a. ford_prefect (homepage, bugs)
PSA: Breaking webrtc-audio-processing API (October 28, 2015, 17:35 UTC)

I know it’s been ages, but I am now working on updating the webrtc-audio-processing library. You might remember this as the code that we split off from the code to use in the PulseAudio echo cancellation module.

This is basically just the AudioProcessing module, bundled as a standalone library so that we can use the fantastic AEC, AGC, and noise suppression implementation from that code base. For packaging simplicity, I made a copy of the necessary code, and wrote an autotools-based build system around that.

Now since I last copied the code, the library API has changed a bit — nothing drastic, just a few minor cleanups and removed API. This wouldn’t normally be a big deal since this code isn’t actually published as external API — it’s mostly embedded in the Chromium and Firefox trees, probably other projects too.

Since we are exposing a copy of this code as a standalone library, this means that there are two options — we could (a) just break the API, and all dependent code needs to be updated to be able to use the new version, or (b) write a small wrapper to try to maintain backwards compatibility.

I’m inclined to just break API and release a new version of the library which is not backwards compatible. My rationale for this is that I’d like to keep the code as close to what is upstream as possible, and over time it could become painful to maintain a bunch of backwards-compatibility code.

A nicer solution would be to work with upstream to make it possible to build the AudioProcessing module as a standalone library. While the folks upstream seemed amenable to the idea when this came up a few years ago, nobody has stepped up to actually do the work for this. In the mean time, a number of interesting features have been added to the module, and it would be good to pull this in to use in PulseAudio and any other projects using this code (more about this in a follow-up post).

So if you’re using webrtc-audio-processing, be warned that the next release will probably break API, and you’ll need to update your code. I’ll try to publish a quick update guide when releasing the code, but if you want to look at the current API, take a look at the current audio_processing.h.

p.s.: If you do use webrtc-audio-processing as a dependency, I’d love to hear about it. As far as I know, PulseAudio is the only user of this library at the moment.

October 22, 2015
Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
WTF Google (October 22, 2015, 13:02 UTC)

Every now and then this happens: I have no idea what it is supposed to do, but it makes the whole site non-interactive, which sucks.
If this is Google's attempt to recruit me, or whatever, it should try to do it with less JavaScript.

Maybe I should just DuckDuckGo instead?

October 21, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Nobody hears you being subtle on twitter (October 21, 2015, 10:57 UTC)

You might be subtle like this or just work on your stuff like that but then nobody will know that you are the one that did something (and praise somebody else completely unrelated for your stuff, e.g. Anton not being praised much for the HEVC threaded decoding, the huge work on ref-counted AVFrame and many other things).

Blogging is boring

Once you wrote something in code talking about it gets sort of boring, the code is there, it works and maybe you spent enough time on the mailing list and irc discussing about it that once it is done you wouldn’t want to think about it for at least a week.

The people at xiph got it right and they wrote awesome articles about what they are doing.

Blogging is important

JB got it right by writing posts about what happened every week. Now journalist can pick from there what’s cool and coming from VLC and not have to try to extract useful information from git log, scattered mailing lists and conversations on irc.
I’m not sure I’ll have the time to do the same, but surely I’ll prod at least Alexandra and the others to write more.

October 20, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
uhashring : consistent hashing in python (October 20, 2015, 21:05 UTC)

It’s been quite some time since I wanted to use a consistent hashing based distribution of my workload in a quite big cluster at work. So when we finally reached the point where this became critical for some job processing I rushed to find out what python library I could use to implement this easily and efficiently.

I was surprised not to find a clear “winner” for such a library. The more “popular” named hash_ring has a rather unoptimized source code and is dead (old issues, no answer). Some others stepped up since but with no clear interest for contributions and almost no added features for real world applications.

So I packaged the ketama C library and its python binding on my overlay to get intimate with its algorithm. Then I started working on my own pure python library and released uhashring on Pypi and on Gentoo portage !


  • instance-oriented usage so you can use your consistent hash ring object directly in your code (see advanced usage).
  • a lot of convenient methods to use your consistent hash ring in real world applications.
  • simple integration with other libs such as memcache through monkey patching.
  • all the missing functions in the libketama C python binding (which is not even available on pypi).
  • another and more performant consistent hash algorithm if you don’t care about the ketama compatibility (see benchmark).
  • native pypy support.
  • tests of implementation, key distribution and ketama compatibility.


Not so sure about hash tables and consistent hashing ? Read my consistent hashing 101 explanation attempt !

October 19, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
Consistent Hashing 101 (October 19, 2015, 14:48 UTC)

When working on distributed systems, we often have to distribute some kind of workload on different machines (nodes) of a cluster so we have to rely on a predictable and reliable key/value mapping algorithm.

If you’re not sure about what that means, just consider the following questions when working on a cluster with a lot of machines (nodes):

  • how could I make sure that all the data for a given user always gets delivered and processed on the same machine ?
  • how could I make sure that I store and query the same cache server for a given key ?
  • how do I split and distribute chunks of a large file across multiple storage machines and then make sure I can still access it through all those machines at once ?

A lot of technologies answer those kind of mapping questions by implementing a hashing based distribution of their workload (be it a distributed processing, file or cache).

Consistent hashing is one of those implementations and I felt like taking some time to explain it further.

Example use case

Here is my attempt to explain what is consistent hashing and why it is needed on distributed systems. To make things fun, we’ll take this simple use case:

  • I have 5 broken cars
  • There are 4 repair factories nearby
  • I have to implement a way to figure out where to send each car to get it fixed
  • I want to ensure that an even number of cars will get fixed on each factory


This gets down to two major questions to solve:

  • what is my selection criteria ? this will be my key.
  • what is the expected answer ? this is my value.

Static mapping

The first approach we could implement is to manually distribute the car based on their colour.

  • key = car’s colour
  • value = factory number

To implement this, you use what we usually call dictionaries on various languages : those are static data structures where you assign a value to a key.

We would then write a mapping of “car color” -> “factory n” and apply this simple rule to decide where to ship a broken car.

  "yellow": "factory 1",
  "orange": "factory 2",
  "red": "factory 3",
  "green": "factory 4",
  "black": "factory 1"

This way we could indeed distribute the car repairs, but we can already see that with that an uneven number of colours ends up in over provisioning the factory 1. But there’s worse:

What if I start getting only yellow broken cars ?

  • I would end up sending all of them to the factory 1 and the other factories would remain almost empty !


This is a serious limitation. We need a dynamic way to calculate the car distribution between the factories, for this we will use a hash algorithm !

Hash tables

A hash table is a data structure where we apply a hash function (algorithm) on the key to compute an index (pointer) into an array of buckets (values) from which we get the value.

MD5 gives very good hashing distribution and is widely available so this makes it a very good candidate for a hashing algorithm.

We can relate it like this to our example :

  • key = car’s plate number
  • hash function = md5
  • array of values = [factory 1, factory 2, factory 3, factory 4]

To find out where we send a car we just could do:

hash = md5(car plate number)
index = int(hash) % size_of(array)
index = 0 if index > size_of(array)
factory = array[index]

In python ? okay !

import md5

factories = [1, 2, 3, 4]

def get_factory(plate):
    hash = int(, 16)
    index = hash % len(factories)
    if index > len(factories):
        index = 0
    return factories[index]

>> 3

>> 3

Wow it’s amazingly simple right ? :)

Now we have a way better car repair distribution !… until something bad happens:

What if a factory burns ?


Our algorithm is based on the number of available factories so removing a factory from our array means that we will redistribute a vast majority of the key mappings from our hash table !

Keep in mind that the more values (factories) you have in your array the worse this problem gets. In our case, given a car’s plate number we are sure that we wouldn’t be able to figure out where a vast majority of them were sent any more.

factories = [1, 2, 4]

>> 2 (was 3 before)

>> 1 (was 3 before)

Even worse is that when factory 3 gets repaired and back in my hash table, I will once again loose track of all my dispatched cars… What we need is a more consistent way of sorting this out.

Consistent hashing

The response to this kind of problem is to implement a consistent hashing algorithm. The goal of this technique is to limit the number of remapped keys when the hash table is resized.

This is possible by imagining our factories as a circle (ring) and the hash of our keys as points on the same circle. We would then select the next factory (value) by going through the circle, always on the same way, until we find a factory.


  • Red hashed plate number would go to factory 1
  • Blue hashed plate number would go to factory 3

This way, when a factory gets added or removed from the ring, we loose only a limited portion of the key/value mappings !

Of course on a real world example we would implement a ring with a lot more of slots by adding the same factories on the ring multiple times. This way the affected range of mappings would be smaller and the impact even more balanced !

For instance, uhashring being fully compatible and defaulting to ketama’s ring distribution, you get 160 points per node on the ring !

I hope I got this little example right and gave you some insight on this very interesting topic !

October 15, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Deprecating AVPicture (October 15, 2015, 19:40 UTC)

In Libav we try to clean up the API and make it more regular, this is one of the possibly many articles I write about APIs, this time about deprecating some relic from the past and why we are doing it.


This struct used to store image data using data pointers and linesizes. It comes from the far past and it looks like this:

typedef struct AVPicture {
    uint8_t *data[AV_NUM_DATA_POINTERS];
    int linesize[AV_NUM_DATA_POINTERS];
} AVPicture;

Once the AVFrame was introduced it was made so it would alias to it and for some time the two structures were actually defined sharing the commond initial fields through a macro.

The AVFrame then evolved to store both audio and image data, to use AVBuffer to not have to do needless copies and to provide more useful information (e.g. the actual data format), now it looks like:

typedef struct AVFrame {
    uint8_t *data[AV_NUM_DATA_POINTERS];
    int linesize[AV_NUM_DATA_POINTERS];

    uint8_t **extended_data;

    int width, height;

    int nb_samples;

    int format;

    int key_frame;

    enum AVPictureType pict_type;

    AVRational sample_aspect_ratio;

    int64_t pts;

} AVFrame;

The image-data manipulation functions moved to the av_image namespace and use directly data and linesize pointers, while the equivalent avpicture became a wrapper over them.

int avpicture_fill(AVPicture *picture, uint8_t *ptr,
                   enum AVPixelFormat pix_fmt, int width, int height)
    return av_image_fill_arrays(picture->data, picture->linesize,
                                ptr, pix_fmt, width, height, 1);

int avpicture_layout(const AVPicture* src, enum AVPixelFormat pix_fmt,
                     int width, int height,
                     unsigned char *dest, int dest_size)
    return av_image_copy_to_buffer(dest, dest_size,
                                   src->data, src->linesize,
                                   pix_fmt, width, height, 1);


It is also used in the subtitle abstraction:

typedef struct AVSubtitleRect {
    int x, y, w, h;
    int nb_colors;

    AVPicture pict;
    enum AVSubtitleType type;

    char *text;
    char *ass;
    int flags;
} AVSubtitleRect;

And to crudely pass AVFrame from the decoder level to the muxer level, for certain rawvideo muxers by doing something such as:   = (uint8_t *)frame;
    pkt.size   =  sizeof(AVPicture);

AVPicture problems

In the codebase its remaining usage is dubious at best:

AVFrame as AVPicture

In some codecs the AVFrame produced or consumed are casted as AVPicture and passed to avpicture functions instead
of directly use the av_image functions.


For the subtitle codecs, accessing the Rect data requires a pointless indirection, usually something like:

    wrap3 = rect->pict.linesize[0];
    p = rect->[0];
    pal = (const uint32_t *)rect->[1];  /* Now in YCrCb! */


Copying memory from a buffer to another when can be avoided is consider a major sin (“memcpy is murder”) since it is a costly operation in itself and usually it invalidates the cache if we are talking about large buffers.

Certain muxers for rawvideo, try to spare a memcpy and thus avoid a “murder” by not copying the AVFrame data to the AVPacket.

The idea in itself is simple enough, store the AVFrame pointer as if it would point a flat array, consider the data size as the AVPicture size and hope that the data pointed by the AVFrame remains valid while the AVPacket is consumed.

Simple and faulty: with the AVFrame ref-counted API codecs may use a Pool of AVFrames and reuse them.
It can lead to surprising results because the buffer gets updated before the AVPacket is actually written.
If the frame referenced changes dimensions or gets deallocated it could even lead to crashes.

Definitely not a great idea.


Vittorio, wm4 and I worked together to fix the problems. Radically.

AVFrame as AVPicture

The av_image functions are now used when needed.
Some pointless copies got replaced by av_frame_ref, leading to less memory usage and simpler code.

No AVPictures are left in the video codecs.


The AVSubtitleRect is updated to have simple data and linesize fields and each codec is updated to keep the AVPicture and the new fields in sync during the deprecation window.

The code is already a little easier to follow now.


Just dropping the “feature” would be a problem since those muxers are widely used in FATE and the time the additional copy takes adds up to quite a lot. Your regression test must be as quick as possible.

I wrote a safer wrapper pseudo-codec that leverages the fact that both AVPacket and AVFrame use a ref-counted system:

  • The AVPacket takes the AVFrame and increases its ref-count by 1.
  • The AVFrame is then stored in the data field and wrapped in a custom AVBuffer.
  • That AVBuffer destructor callback unrefs the frame.

This way the AVFrame data won’t change until the AVPacket gets destroyed.

Goodbye AVPicture

With the release 14 the AVPicture struct will be removed completely from Libav, people using it outside Libav should consider moving to use full AVFrame (and leverage the additional feature it provides) or the av_image functions directly.

October 10, 2015
Robin Johnson a.k.a. robbat2 (homepage, bugs)

Some mis-reading the help for 'rbd resize' recently lead somebody to making an RBD volume extremely large, if the units are bytes, this command makes sense:
rbd resize --size $((155*1024*1024*1024)) $RBD_VOL_NAME
But, as it happens, the units are actually in MiB.

# rbd info $RBD_VOL_NAME
rbd image '$RBD_VOL_NAME':
 size 155 PB in 41607495680 objects
Oops, that's a bit too big, and I doubt your cluster has that much space for a single volume, even if it's sparsely allocated.

There is the --allow-shrink option to rbd resize, however on a volume this large, at least in the Hammer release, it will pretty much never return (it eventually times out).

Future work for RBD resize might include a much more intelligent resize command, that checks the highest non-null block, perhaps a reverse iterator looking for keys named at most the size of the device.
Making resize ask for confirmation with output of the new size, or have a dry-run option would also be helpful to avoid this problem.

So, as a workaround, I give you the following useful script, that manually changes the size omap field directly. It is wildly unsafe if you are already using anything past the target size, but it's fine otherwise.
You need to ensure nothing is using the volume when you apply this...

# Prerequisite: From Infernalis source or newer, src/pybind/, src/pybind/ 
# Make sure you have this fix:
from rados import Rados, ReadOpCtx, WriteOpCtx
import rbd
import struct
import ctypes
RBD_NAME = '...'
NEWSIZE = 155 * 1024 * 1024 * 1024 # 155 GiB

with Rados(conffile='/etc/ceph/ceph.conf') as r:
  print 'RADOS object', r
  print 'RADOS version', r.version()
  print 'Cluster Stats', r.get_cluster_stats()
  with r.open_ioctx('rbd') as ioctx:
    with rbd.Image(ioctx, RBD_NAME, read_only=True) as img:
      imgstat = img.stat()
      print imgstat
      header_name = imgstat['block_name_prefix'].replace('rbd_data.', 'rbd_header.', 1)

    with ReadOpCtx(ioctx) as read_op:
      kv_it = ioctx.get_omap_vals(read_op, "", "", 500)
      ioctx.operate_read_op(read_op, header_name)
      kv = dict(kv_it[0])
      print 'OMAP of', header_name, kv
    newkeys = ('size', )
    newvals = (struct.pack('<Q', NEWSIZE), ) # unsigned long long, little endian
    with WriteOpCtx(ioctx) as write_op:
      ioctx.set_omap(write_op, newkeys, newvals)
      ioctx.operate_write_op(write_op, header_name)
    with rbd.Image(ioctx, RBD_NAME, read_only=True) as img:
      imgstat = img.stat()
      print imgstat


October 09, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)


My tech articles—especially Linux ones—are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Yet again, I had troubles with restarting amavisd after some package updates. You can see about the other problems that I’ve had with the application in these two previous blog posts: missing SpamAssassin rules causing amavis to fail and start-up failures due to a missing Perl IPv6 module.

This time, the problem was related to the backend database used for Bayes filtering within SpamAssassin, although it wasn’t readily clear what the problem was. After updating SpamAssassin, I saw the following message from Portage (Gentoo‘s package manager):

* You need to configure your database to be able to use Bayes filter
* with database backend, otherwise it will still use (and need) the
* Berkeley DB support.
* Look at the sql/README.bayes file in the documentation directory
* for how to configure it.

At first, I thought that there was a problem with my MariaDB installation, but then I remembered that I’m currently still using BerkeleyDB (BDB) for SpamAssassin.

After trying to stop and restart amavisd, zapping the configuration (which was a problem mentioned in one of my previous posts), manually killing the running amavisd processes, and a few other things, I was still left with a configuration that would cause amavisd to fail on startup:

* Starting amavisd-new ...
Problem in Amavis::DB or Amavis::DB::SNMP code:
BerkeleyDB needs compatible versions of libdb & db.h
you have db.h version 6.0.30 and libdb version 6.0.35
Compilation failed in require at (eval 93) line 20.
BEGIN failed--compilation aborted at (eval 93) line 20. [ !! ]
* ERROR: amavisd failed to start

Out of sheer desperation, I attempted what seemed to be an unlikely fix by recompiling BerkleyDB (via emerge -v dev-perl/BerkeleyDB). Lo and behold, amavis decided to start after that recompilation! It’s a bit strange to me that the standard tools like revdep-rebuild and emerge @preserved-rebuild didn’t catch this problem.

In any case, third time is a charm with amavisd. Hopefully there won’t be any further problems when attempting to reload or restart it.


October 04, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs)


Frequent updates are important, especially with Gentoo. You probably have seen conflicts that filled several screens and were not trivial to resolve. With frequent updates, handling conflicts becomes a lot easier. Frequent updates also gets you security fixes earlier. I would have done it from the beginning, if something would have reminded me to update. By now, for notifications on available updates I use

Former is a tool by Benedikt Böhm sending e-mails with emerge --pretend output whenever it changes. Latter is a bit of a quick hack by me that sits in the tray and shows an icon if updates are available with the number of updates in a tooltip. It requires a cron job to run something like eix-sync once a day but that’s probably a good idea anyway.
Maybe it’s something that you want to use too. If you run into bugs, please report them and share your patches if any. GitHub allows attaching text files by now, it may work with patches, too.

I would be curious to hear what other tools you use for being notified of updates on Gentoo. Please leave a comment below.

Best, Sebastian

Latest Toy: The Orange Pi PC (October 04, 2015, 14:33 UTC)

You may have heard about the latest Raspberry Pi 2 clone: The Orange Pi PC, a device that sells for only $15 yet comes with a quad-core Allwinner H3 and 1 GB RAM. (But don't be fooled - while "open source" is mentioned everywhere on that website, the actual open source situation is still pretty bad).

After my Raspberry Pi started acting up and run very unstable a year ago, this looked like a nice replacement. Thus I decided to order the "Orange Pi PC set 3" including a plastic case and power cable from AliExpress, and will try to get Gentoo running on that device. The manufacturer's Gentoo forum section is all crickets, so I expect to encounter at least some problems.

If you want to see this thing in action, or at least watch me struggling to get Gentoo running, come next weekend (10+11 October 2015) to the Prague Linux Days and visit the Gentoo booth.

October 02, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Decoupling an API – Part II (October 02, 2015, 14:39 UTC)

During the VDD we had lots of discussions and I enjoyed reviewing the initial NihAV implementation. Kostya already wrote some more about the decoupled API that I described at high level here.

This article is about some possible implementation details, at least another will follow.

The new API requires some additional data structures, mainly something to keep the data that is being consumed/produced, additional implementation-callbacks in AVCodec and possibly a mean to skip the queuing up completely.

Data Structures

AVPacketQueue and AVFrameQueue

In the previous post I considered as given some kind of Queue.

Ideally the API for it could be really simple:

typedef struct AVPacketQueue;

AVPacketQueue *av_packet_queue_alloc(int size);
int av_packet_queue_put(AVPacketQueue *q, AVPacket *pkt);
int av_packet_queue_get(AVPacketQueue *q, AVPacket *pkt);
int av_packet_queue_size(AVPacketQueue *q);
void av_packet_queue_free(AVPacketQueue **q);
typedef struct AVFrameQueue;

AVFrameQueue *av_frame_queue_alloc(int size);
int av_frame_queue_put(AVFrameQueue *q, AVPacket *pkt);
int av_frame_queue_get(AVFrameQueue *q, AVPacket *pkt);
int av_frame_queue_size(AVFrameQueue *q);
void av_frame_queue_free(AVFrameQueue **q);

Internally it leverages the ref-counted API (av_packet_move_ref and av_frame_move_ref) and any data structure that could fit the queue-usage. It will be used in a multi-thread scenario so a form of Lock has to be fit into it.

We have already something specific for AVPlay, using a simple Linked List and a FIFO for some other components that have a near-constant maximum number of items (e.g. avconv, NVENC, QSV).

Possibly also a Tree could be used to implement something such as av_packet_queue_insert_by_pts and have some forms of reordering happen on the fly. I’m not a fan of it, but I’m sure someone will come up with the idea..

The Queues are part of AVCodecContext.

typedef struct AVCodecContext {
    // ...

    AVPacketQueue *packet_queue;
    AVFrameQueue *frame_queue;

    // ...
} AVCodecContext;

Implementation Callbacks

In Libav the AVCodec struct describes some specific codec features (such as the supported framerates) and hold the actual codec implementation through callbacks such as init, decode/encode2, flush and close.
The new model obviously requires additional callbacks.

Once the data is in a queue it is ready to be processed, the actual decoding or encoding can happen in multiple places, for example:

  • In avcodec_*_push or avcodec_*_pull, once there is enough data. In that case the remaining functions are glorified proxies for the matching queue function.
  • somewhere else such as a separate thread that is started on avcodec_open or the first avcodec_decode_push and is eventually stopped once the context related to it is freed by avcodec_close. This is what happens under the hood when you have certain hardware acceleration.


typedef struct AVCodec {
    // ... previous fields
    int (*need_data)(AVCodecContext *avctx);
    int (*has_data)(AVCodecContext *avctx);
    // ...
} AVCodec;

Those are used by both the encoder and decoder, some implementations such as QSV have functions that can be used to probe the internal state in this regard.


typedef struct AVCodec {
    // ... previous fields
    int (*decode_push)(AVCodecContext *avctx, AVPacket *packet);
    int (*decode_pull)(AVCodecContext *avctx, AVFrame *frame);
    // ...
} AVCodec;

Those two functions can take a portion of the work the current decode function does, for example:
– the initial parsing and dispatch to a worker thread can happen in the _push.
– reordering and blocking until there is data to output can happen on _pull.

Assuming the reordering does not happen outside the pull callback in some generic code.


typedef struct AVCodec {
    // ... previous fields
    int (*encode_push)(AVCodecContext *avctx, AVFrame *frame);
    int (*encode_pull)(AVCodecContext *avctx, AVPacket *packet);
} AVCodec;

As per the Decoding callbacks, encode2 workload is split. the _push function might just keep queuing up until there are enough frames to complete the initial the analysis, while, for single thread encoding, the rest of the work happens at the _pull.

Yielding data directly

So far the API mainly keeps some queue filled and let some magic happen under the hood, let see some usage examples first:

Simple Usage

Let’s expand the last example from the previous post: register callbacks to pull/push the data and have some simple loops.


typedef struct DecodeCallback {
    int (*pull_packet)(void *priv, AVPacket *pkt);
    int (*push_frame)(void *priv, AVFrame *frame);
    void *priv_data_pull, *priv_data_push;
} DecodeCallback;

Two pointers since you pull from a demuxer+parser and you push to a splitter+muxer.

int decode_loop(AVCodecContext *avctx, DecodeCallback *cb)
    AVPacket *pkt  = av_packet_alloc();
    AVFrame *frame = av_frame_alloc();
    int ret;
    while ((ret = avcodec_decode_need_data(avctx)) > 0) {
        ret = cb->pull_packet(cb->priv_data_pull, pkt);
        if (ret < 0)
            goto end;
        ret = avcodec_decode_push(avctx, pkt);
        if (ret < 0)
            goto end;
    while ((ret = avcodec_decode_have_data(avctx)) > 0) {
        ret = avcodec_decode_pull(avctx, frame);
        if (ret < 0)
            goto end;
        ret = cb->push_frame(cb->priv_data_push, frame);
        if (ret < 0)
            goto end;

    return ret;


For encoding something quite similar can be done:

typedef struct EncodeCallback {
    int (*pull_frame)(void *priv, AVFrame *frame);
    int (*push_packet)(void *priv, AVPacket *packet);
    void *priv_data_push, *priv_data_pull;
} EncodeCallback;

The loop is exactly the same beside the data types swapped.

int encode_loop(AVCodecContext *avctx, EncodeCallback *cb)
    AVPacket *pkt  = av_packet_alloc();
    AVFrame *frame = av_frame_alloc();
    int ret;
    while ((ret = avcodec_encode_need_data(avctx)) > 0) {
        ret = cb->pull_frame(cb->priv_data_pull, frame);
        if (ret < 0)
            goto end;
        ret = avcodec_encode_push(avctx, frame);
        if (ret < 0)
            goto end;
    while ((ret = avcodec_encode_have_data(avctx)) > 0) {
        ret = avcodec_encode_pull(avctx, pkt);
        if (ret < 0)
            goto end;
        ret = cb->push_packet(cb->priv_data_push, pkt);
        if (ret < 0)
            goto end;

    return ret;


Transcoding, the naive way, could be something such as

int transcode(AVFormatContext *mux,
              AVFormatContext *dem,
              AVCodecContext *enc,
              AVCodecContext *dec)
    DecodeCallbacks dcb = {
        dem, enc->frame_queue };
    EncodeCallbacks ecb = {
        enc->frame_queue, mux };
    int ret = 0;

    while (ret > 0) {
        if ((ret = decode_loop(dec, &dcb)) > 0)
            ret = encode_loop(enc, &ecb);

One loop feeds the other throught the queue. get_packet and push_packet are muxing and demuxing functions, they might end up being other two Queue functions once the AVFormat layer gets a similar overhaul.

Advanced usage

From the examples above you would notice that in some situation you would possibly do better,
all the loops pull data from a queue push it immediately to another:

  • why not feeding right queue immediately once you have the data ready?
  • why not doing some processing before feeding the decoded data to the encoder, such as conver the pixel format?

Here some additional structures and functions to enable advanced users:

typedef struct AVFrameCallback {
    int (*yield)(void *priv, AVFrame *frame);
    void *priv_data;
} AVFrameCallback;

typedef struct AVPacketCallback {
    int (*yield)(void *priv, AVPacket *pkt);
    void *priv_data;
} AVPacketCallback;

typedef struct AVCodecContext {
// ...

AVFrameCallback *frame_cb;
AVPacketCallback *packet_cb;

// ...

} AVCodecContext;

int av_frame_yield(AVFrameCallback *cb, AVFrame *frame)
    return cb->yield(cb->priv_data, frame);

int av_packet_yield(AVPacketCallback *cb, AVPacket *packet)
    return cb->yield(cb->priv_data, packet);

Instead of using directly the Queue API, would be possible to use yield functions and give the user a mean to override them.

Some API sugar could be something along the lines of this:

int avcodec_decode_yield(AVCodecContext *avctx, AVFrame *frame)
    int ret;

    if (avctx->frame_cb) {
        ret = av_frame_yield(avctx->frame_cb, frame);
    } else {
        ret = av_frame_queue_put(avctx->frame_queue, frame);

    return ret;

Whenever a frame (or a packet) is ready it could be passed immediately to another function, depending on your threading model and cpu it might be much more efficient skipping some enqueuing+dequeuing steps such as feeding directly some user-queue that uses different datatypes.

This approach might work well even internally to insert bitstream reformatters after the encoding or before the decoding.

Open problems

The callback system is quite powerful but you have at least a couple of issues to take care of:
– Error reporting: when something goes wrong how to notify what broke?
– Error recovery: how much the user have to undo to fallback properly?

Probably this part should be kept for later, since there is already a huge amount of work.

What’s next

Muxing and demuxing

Ideally the container format layer should receive the same kind of overhaul, I’m not even halfway documenting what should
change, but from this blog post you might guess the kind of changes. Spoiler: The I/O layer gets spun in a separate library.

Proof of Concept

Soon^WNot so late I’ll complete a POC out of this and possibly hack avplay so that either it uses QSV or videotoolbox as test-case (depending on which operating system I’m playing with when I start), probably I’ll see which are the limitations in this approach soon.

If you like the ideas posted above or you want to discuss them more, you can join the Libav irc channel or mailing list to discuss and help.

October 01, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Tonight was an outstanding evening for me, as I got to experience The Iron Giant in the theatre again, just like I did 16 years ago in 1999. The film has been enhanced, and it was brought back to theatres across the United States by Fathom Events for one night only: 30 September 2015 starting at 19:00 local time.

The Iron Giant Signature Edition ticket stub
The Iron Giant: Signature Edition ticket stub

If you haven’t had the honour of seeing Brad Bird’s (Animator of The Fox and the Hound, and writer/director of some greats like The Incredibles, and Ratatouille), then please do yourself a favour, and find a copy (either an older version to buy, or even stream it). It’s a truly remarkable film that spoke to me when I first saw it as an adolescent years and years ago.

Back in 1999 and for some years to come, I couldn’t get enough of The Iron Giant because I loved the hand-animated artwork, the incredible story, and some of the great one-liners and comedic scenes. Thereafter, it slipped to the back of my mind, but was never really forgotten. Then, several years ago, I got the joy of reliving it and experiencing it with a really great kid. It took on a completely new meaning to me when I saw the remarkable lessons that it taught—love conquering fear, being who you are, and doing everything within your power to help people.

The Iron Giant Signature Edition ticket stub
The Iron Giant and Hogarth Hughes
(Click to enlarge)

This time around, I got to see it (and introduce it) to my best friend in the world. She also enjoyed it (but, of course, I knew that she would). :-) Before the film started, they were showing various trivia questions about it (like in the screenshot above). I knew most of them, but this one in particular was really cool. By the way, the answer was Ted Hughes, which made Hogarth’s name even more significant. All in all, I can’t think of a better way to spend an evening than with a great friend, a fantastic film, and a hefty dose of nostalgia.


September 29, 2015
Paweł Hajdan, Jr. a.k.a. phajdan.jr (homepage, bugs)
Gentoo's top-notch user community (September 29, 2015, 19:39 UTC)

One of the best parts of Gentoo for me is the community.

For example, I regularly receive email from people who volunteer to help with testing hard masked www-client/chromium ebuilds. FWIW you don't need to email me or the Gentoo Chromium team - just start testing and filing bugs. On the other hand, I do appreciate when people express interest in helping out.

Another example is helping with getting bugs resolved.

Bug #556812 looked rather mysterious - until Guillaume ZITTA found that "build" is a red herring, and in fact the "tracing" module being imported is a different one (from dev-python/tracing as opposed to chromium sources). It was an unfortunate names collision - once found, quite easy to fix.

In bug #553502 Johan Hovold found we need to require postproc USE flag for libvpx to avoid a crash.

See bug #551666 for some Alexey Dobriyan's gdb magic mostly based on a single line segfault report from dmesg...

These are just a few examples.

By the way, the area where we could use more help is arm architecture support. Some specific bugs where help is wanted are #555702#545368#520180, and #483130. Please report whether you can reproduce or not, post emerge --info from your system and the compressed build log in case build fails.

September 27, 2015
Domen Kožar a.k.a. domen (homepage, bugs)
Friends sometimes let friends curl to shell (September 27, 2015, 17:50 UTC)

Every now and then (actually quite often), people complain on twitter they're afraid of our simple bash installer for Nix package manager:

$ bash <(curl

Example (from today):

There are popular blog posts discouraging use of it.

Ask yourself a question, how would package manager install itself? Via another package manager?

If we assume is not compromised (which is really hard to detect), using TLS to secure connection and with our simple trick to prevent partial download execution (you haven't read the script yet, right?), what can really go wrong?

It's the most transparent way to see how the package manager can be bootstrapped: read the source, Luke.

If you still have a reason why piping to shell is a bad idea, let me know.

September 24, 2015
Matthew Thode a.k.a. prometheanfire (homepage, bugs)
OpenStack on Gentoo is awesome (September 24, 2015, 05:00 UTC)

Some may wonder why to run OpenStack on Gentoo, it's akin to running one extremely complex piece of software on another potentially extremely complex operating system.

I propose Gentoo as the most correct operating system to run OpenStack as Gentoo is best prepared to handle some of the complexities that come with OpenStack.

Things Gentoo does well for OpenStack

  • Dependency resolution - we can take upstream's dependencies (requirements.txt) and map it directly to Gentoo's packages without changing the versions supported. This is not always the case in other distributions.
  • Because of this we can better rely on upstream's testing as validation that OpenStack on Gentoo will work properly.
  • useflags - some dependencies (such as memcached, qemu, etc) are optional depending on what services you are running.
  • python3 support - as upstream is finally moving to python3 we are easily able to extend support as it occurs.
  • patching - allows users to patch anything as needed.
  • Upstream vanilla Openstack - We are not carrying many (I can count them on one hand) patches that are not upstreamed.
  • Testing - All the major services have testing support to validate good installs and / or custom patches not breaking things.

Future of OpenStack on Gentoo

  1. The Liberty release 15/10/2015
    • Upstream is reversioning the major components of OpenStack, 2015.2.0 will not exist, it will be something like 12.0.0.
    • The reversioning will mean some manual intervention if you wish to use Liberty on Gentoo, namely you will need to mask greater than version 2000 of the major components of OpenStack
  2. More services
    • I will at least add Heat during the next (Liberty) release cycle, possibly more
    • I will investigate readding Horizon, but doubt it'll be easily packagable.
  3. Security
    • I'd like to look into some of the selinux policies that are being developed for some of the services. I know nova/qemu has apparmor support.

Limits of testing

You cannot directly test OpenStack packages by emerging with USE="test" FEATURES="test" because there are interdepenecies that cause loops in the depgraph of portage for the (test) dependencies. You can get around it one way though.

# install only the test deps
emerge --onlydeps --oneshot --with-test-deps sys-cluster/nova
# test and install the actual package
USE="test" FEATURES="test" emerge sys-cluster/nova

September 23, 2015
Alex Legler a.k.a. a3li (homepage, bugs)
Repackaging (September 23, 2015, 18:59 UTC)

Update: The site is now live at!

Gentoo has seen quite some refreshing updates this year: In April, our website was relaunched (twice ;)) and our main package repository is now finally powered by git. There is one service though that sort-of relates to both, but this service has not seen an update in quite some time: Let’s talk about

The fact that we now use git for what was before called gentoo-x86  requires a set of changes to our package website. Several repository URLs are different now, and especially the changelogs that are now taken from git log messages need new code to display properly. As many participants in our website survey a while back have also noted, the time is most ripe for a change of scenery on “pgo”.

As having a good package search is one of the most-requested feature for the site, I’ve built a new site around elasticsearch (which also powers the new Gentoo mailing list archives). It is nearing completion as a minimum viable product, but already available for you to test in a beta state.

So, head over to and feel free to browse around. Some things are not available yet and the caching has not been fully enabled, so things might take a second to render. I’d be glad to hear your thoughts and ideas on the site—either as a comment here, or if you prefer on freenode in #gentoo-www.

As a sneak peek, here’s the site rendered on a small screen in case you don’t have your $device handy:

Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

Today's good news is that our manuscript "Liquid-induced damping of mechanical feedback effects in single electron tunneling through a suspended carbon nanotube" has been accepted for publication in Applied Physics Letters. So what's it about?
One of the surprises that suspended, clean carbon nanotubes have in store is that they can start vibrating strongly at millikelvin temperatures without any applied radio-frequency driving signal. This was proposed theoretically several years ago by Usmani et al., as a strong feedback between the transversal vibration of the nanotube and the single electron tunneling through it. The effect was identified in measurements, and for example in a previous publication we have shown that damping induced by a magnetic field can suppress it.
Here, we demonstrate how one and the same device behaves distinctly different depending on the environment medium (or lack of the latter): we compare measurements made at the same temperature in a conventional dilution refrigerator, where the chip is placed into a vacuum chamber, and in a so-called top-loading dilution refrigerator, where the chip is inserted into the 3He/4He liquid of the mixing chamber. The overall electronic properties of the device do not change much, even though the thermal cycling could cause a lot of damage and has done so in the past for other devices. We can here even extract a rough estimate of the liquid helium dielectric constant by comparing the slightly shifted Coulomb oscillation positions of the two measurements.
However, a striking difference appears when looking at finite bias conductance and the mechanical feedback effects. In the viscous helium liquid, the resonator is damped and the vibrations are suppressed, and the unperturbed electronic transport spectrum emerges. Such an inert, liquid environment can thus be used to do transport spectroscopy at high transparency of the tunnel barriers and high applied bias voltages - parameter regions interesting for e.g. non-equilibrium Kondo phenomena, where otherwise mechanically-induced features would make data evaluation highly challenging.

"Liquid-induced damping of mechanical feedback effects in single electron tunneling through a suspended carbon nanotube"
D. R. Schmid, P. L. Stiller, Ch. Strunk, and A. K. Hüttel
Applied Physics Letters 107, 123110 (2015); arXiv:1407.2114 (PDF)

September 20, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Switching focus at work (September 20, 2015, 11:29 UTC)

Since 2010, I was at work responsible for the infrastructure architecture of a couple of technological domains, namely databases and scheduling/workload automation. It brought me in contact with many vendors, many technologies and most importantly, many teams within the organization. The focus domain was challenging, as I had to deal with the strategy on how the organization, which is a financial institution, will deal with databases and scheduling in the long term.

This means looking at the investments related to those domains, implementation details, standards of use, features that we will or will not use, positioning of products and so forth. To do this from an architecture point of view means that I not only had to focus on the details of the technology and understand all their use, but also become a sort-of subject matter expert on those topics. Luckily, I had (well, still have) great teams of DBAs (for the databases) and batch teams (for the scheduling/workload automation) to keep things in the right direction.

I helped them with a (hopefully sufficiently) clear roadmap, investment track, procurement, contract/terms and conditions for use, architectural decisions and positioning and what not. And they helped me with understanding the various components, learn about the best use of these, and of course implement the improvements that we collaboratively put on the roadmap.

Times, they are changing

Last week, I flipped over a page at work. Although I remain an IT architect within the same architecture team, my focus shifts entirely. Instead of a fixed domain, my focus is now more volatile. I leave behind the stability of organizationally anchored technology domains and go forward in a more tense environment.

Instead of looking at just two technology domains, I need to look at all of them, and find the right balance between high flexibility demands (which might not want to use current "standard" offerings) which come up from a very agile context, and the almost non-negotionable requirements that are typical for financial institutions.

The focus is also not primarily technology oriented anymore. I'll be part of an enterprise architecture team with direct business involvement and although my main focus will be on the technology side, it'll also involve information management, business processes and applications.

The end goal is to set up a future-proof architecture in an agile, fast-moving environment (contradictio in terminis ?) which has a main focus in data analytics and information gathering/management. Yes, "big data", but more applied than what some of the vendors try to sell us ;-)

I'm currently finishing off the high-level design and use of a Hadoop platform, and the next focus will be on a possible micro-service architecture using Docker. I've been working on this Hadoop design for a while now (but then it was together with my previous function at work) and given the evolving nature of Hadoop (and the various services that surround it) I'm confident that it will not be the last time I'm looking at it.

Now let me hope I can keep things manageable ;-)

Peter Wilmott a.k.a. p8952 (homepage, bugs)
Thirty three seconds on middle street (September 20, 2015, 00:00 UTC)

September 19, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs)

A bit long but very interesting…

Presence: Bringing Your Boldest Self to Your Biggest Challenges

September 14, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)

In my last post, I explained the new hope we had in following some newly added recommended steps before trying to migrate our production cluster to mongoDB 3.0 WiredTiger.

The most demanding step was migrating all our production servers data storage filesystems to XFS which obviously required a resync of each node… But we ended up being there pretty fast and were about to try again as 3.0.5 was getting ready, until we saw this bug coming !

I guess you can understand why we decided to wait for 3.0.6… which eventually got released with a more peaceful changelog this time.

The 3.0.6 crash test

We decided as usual to test the migration to WiredTiger in two phases.

  1. Migrate all our secondaries to the WiredTiger engine (full resync). Wait a week to see if this has any effect on our cluster.
  2. Switch all the MMapv1 primary nodes to secondary and let our WiredTiger secondary nodes become the primary nodes of our cluster. Pray hard that this time it will not break under our workload.

Step 1 results were good, nothing major changed and even our mongo dumps were still working this time (yay!). One week later, everything was still working smoothly.

Step 2 was the big challenge which failed horribly last time. Needless to say that we were quite stressed when doing the switch. But it worked smoothly and nothing broke + performances gains were huge !

The results

Nothing speaks better than metrics, so I’ll just comment them quickly as they speak by themselves. I obviously can’t disclose the scales sorry.

Insert-only operations gained 25x performance









Upsert-heavy operations gained 5x performance









Disk I/O also showed mercy to the disk overall usage. This is due to WiredTiger superior caching and disk flushing mechanisms.








Disk usage decreased dramatically thanks to WiredTiger compression






The last and next step

As of today, we still run our secondaries with the MMapv1 engine and are waiting a few weeks to see if anything goes wrong in the long run. Shall we need to roll back, we’d be able to do so very easily.

Then when we get enough uptime using WiredTiger, we will make the final switch to a full Roarring production cluster !

Sven Vermeulen a.k.a. swift (homepage, bugs)
Getting su to work in init scripts (September 14, 2015, 14:37 UTC)

While developing an init script which has to switch user, I got a couple of errors from SELinux and the system itself:

~# rc-service hadoop-namenode format
Authenticating root.
 * Formatting HDFS ...
su: Authentication service cannot retrieve authentication info

The authentication log shows entries such as the following:

Sep 14 20:20:05 localhost unix_chkpwd[5522]: could not obtain user info (hdfs)

I've always had issues with getting su to work properly again. Now that I have what I think is a working set, let me document it for later (as I still need to review why they are needed):

# Allow initrc_t to use unix_chkpwd to check entries
# Without it gives the retrieval failure

# Allow initrc_t to query selinux access, otherwise avc assertion
allow initrc_t self:netlink_selinux_socket { bind create read };

# Allow initrc_t to honor the pam_rootok setting
allow initrc_t self:passwd { passwd rootok };

With these SELinux rules, switching the user works as expected from within an init script.

September 10, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Custom CIL SELinux policies in Gentoo (September 10, 2015, 05:13 UTC)

In Gentoo, we have been supporting custom policy packages for a while now. Unlike most other distributions, which focus on binary packages, Gentoo has always supported source-based packages as default (although binary packages are supported as well).

A recent commit now also allows CIL files to be used.

Policy ebuilds, how they work

Gentoo provides its own SELinux policy, based on the reference policy, and provides per-module ebuilds (packages). For instance, the SELinux policy for the screen package is provided by the sec-policy/selinux-screen package.

The package itself is pretty straight forward:

# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id$


inherit selinux-policy-2

DESCRIPTION="SELinux policy for screen"

if [[ $PV == 9999* ]] ; then
        KEYWORDS="~amd64 ~x86"

The real workhorse lays within a Gentoo eclass, something that can be seen as a library for ebuilds. It allows consolidation of functions and activities so that a large set of ebuilds can be simplified. The more ebuilds are standardized, the more development can be put inside an eclass instead of in the ebuilds. As a result, some ebuilds are extremely simple, and the SELinux policy ebuilds are a good example of this.

The eclass for SELinux policy ebuilds is called selinux-policy-2.eclass and holds a number of functionalities. One of these (the one we focus on right now) is to support custom SELinux policy modules.

Custom SELinux policy ebuilds

Whenever a user has a SELinux policy that is not part of the Gentoo policy repository, then the user might want to provide these policies through packages still. This has the advantage that Portage (or whatever package manager is used) is aware of the policies on the system, and proper dependencies can be built in.

To use a custom policy, the user needs to create an ebuild which informs the eclass not only about the module name (through the MODS variable) but also about the policy files themselves. These files are put in the files/ location of the ebuild, and referred to through the POLICY_FILES variable:

# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id$

POLICY_FILES="oracle.te oracle.if oracle.fc"

inherit selinux-policy-2

DESCRIPTION="SELinux policy for screen"

if [[ $PV == 9999* ]] ; then
        KEYWORDS="~amd64 ~x86"

The eclass generally will try to build the policies, converting them into .pp files. With CIL, this is no longer needed. Instead, what we do is copy the .cil files straight into the location where we place the .pp files.

From that point onwards, managing the .cil files is similar to .pp files. They are loaded with semodule -i and unloaded with semodule -r when needed.

Enabling CIL in our ebuilds is a small improvement (after the heavy workload to support the 2.4 userspace) which allows Gentoo to stay ahead in the SELinux world.

Luca Barbato a.k.a. lu_zero (homepage, bugs)
Libav: The project (September 10, 2015, 02:51 UTC)

This is a tiny introduction to Libav, the organization.


Libav is an informal flat organization focused on developing libraries
and tools to decode, manipulate and encode multimedia contents.

The project tries to be as non-hierarchical as possible, every contributor must abide by a well defined set of rules, no matter which are the roles taken.

The project aims to provide useful tools, written in portable code that is readable, trustworthy and performant.

Decisions are agreed with through near-unanimous consensus, discussions may happen over the irc, mailing-list or real life meetings.

If possible, conflicts should be avoided and otherwise resolved.


The project strives to provide a pleasant environment for everybody.

Every contributor is considered a member of the team, does not matter if he or she is a newcomer or a founder: nobody has special rights or prerogatives.

Well defined rules had been adopted since the foundation to ensure fairness.

Code of Conduct

A quite simple code of conduct is enforced over all the communication media.

It boils down to respecting the other people and being pleasant to deal with.

It is commonly enforced with a friendly warning, followed by the request to leave if the person is unable to behave and, then, eventual removal if anything else fails.

Over the history of the project a single person managed to reach the last step.


The project has simple workflow:

  • Every patch must be sent to the mailing-list
  • Every patch must get a review and an Ok before it lands in the master branch.

Code Quality

Plenty of documentation on how to prepare good patches is provided.

The reviewers usually help newcomers by reformatting the first patches and pointing and fixing common pitfalls.

If some mistakes are not caught during the review, there are few additional means to prevent them from hitting a release.


Even if the project is non-hierarchical, is possible to define specific roles within it.

Roles such as committer or infrastructure admin do not really give additional power, just additional responsabilities.


Contributing to Libav makes you a Contributor!
Anybody who reviews patches, writes patches, helps triaging bugs, writes documentation, helps people having problems, keeps our infrastructure running is considered a contributor.

Does not matter how little, any help is welcome.

It does not come with special perks beside maybe receive some branded clothing and chocolate during events.


Many eyes might not make every bug shallow, but probably a second and a third pair might prevent some silly mistakes.

Because of our workflow, spending time reading other people patches is quite common.

People with specific expertise might get nagged to give their opinion more than others, but everybody might spot something that looks wrong and probably is.


Checking that the bugs are fixed and ask for better reports is important.

Traditionally most of the issues are reported and fixed in real-time on irc.

Bugs reported in the issue tracker had been neglected more often than not since fixing bugs reported interactively usually catch the attention and gain priority, I set up a dedicated bot broadcasting the new bugs on irc and it somehow helped to improve the situation.

Bug wrangling is something that is much appreciated and just few people devote time on.


Nobody can push a patch to the master until it is reviewed, but somebody has to push it once it is.

Being a committer requires you to take the reviewed patches, make sure they work as expected either locally
or pushing them through the Oracle and possibly do minor edits such as removing typos from the commit messages.

Patches from a committer do still need a review from a third party.

Infrastructure Administrator

Libav does use quite a bit of hardware to test the code over the various fate instances.

Not to mention the machines used to keep the git repository, the samples collection, the website, the patch trackers, the wiki and the issue tracker.

Most of it comes from people devoting their time (and beside few exceptions) their own hardware, definitely this role requires a huge amount of dedication.

Join us!

If you ended up reading until here and you would like to contribute to the project please contact us.

Help is always welcome.

Post Scriptum

This post tries to summarize the project and its structure as if the legends surrounding it do not exist and the project is just a clean slate. Shame on me for not having written this blog post 5 years ago.

Past and Present

I already wrote about the past and the current situation of Libav, if you are curious please do read the previous posts. I will probably blog again about the social issues soon.


The Release 12 is in the ABI break window now and soon the release branch will be spun off! After that some of my plans to improve the API will see some initial implementations and hopefully will be available as part of the release 13 (and nihav)

I will discuss avframe_yield first since Kostya already posted about a better way to handle container formats.

Peter Wilmott a.k.a. p8952 (homepage, bugs)
Experimenting with Astrophotography (September 10, 2015, 00:00 UTC)

September 09, 2015

Recently a friend of mine set up a new audio system and decided to go for one of the popular Sonos alternatives. Helping him setting it up brought out a few interesting questions, some of which I'll try to elaborate on in this post.

This won't be a comprehensive discussion of the developments of the Internet of Things (IoT), that would result in a book rather than a blog post, and several articles have been written about the subject already, including this pc world article that sums up a few elements quite succinctly;

Vint Cerf is known as a "father of the Internet," and like any good parent, he worries about his offspring -- most recently, the IoT.

"Sometimes I'm terrified by it," he said in a news briefing Monday at the Heidelberg Laureate Forum in Germany. "It's a combination of appliances and software, and I'm always nervous about software -- software has bugs."

And that brings a nice introduction to one of the elements of the issue. Software have bugs, and some (actually a lot) are affecting security. This requires, on the onset, three things;

  1. Software vendors needs to be alerted of the vulnerabilities and fix them
  2. Users needs to be have a way to properly update their systems in a way that provide integrity control and authentication (digital signatures)
  3. Users actually have to upgrade their systems.

As we have seen as recently as with the Stagefright vulnerability affecting mobile phones, there is a failure on several of these levels even for a popular operating system such as Android. These failures stems from multiple sources, one of which is that cellphone vendors don't use the latest versions of the OS available across all phones they sell.

There are reasons for this, mainly that the software shipped is often heavily modified with proprietary code to support the specific features of a specific phone, and requirements of modern OSes might not work on older phones due to resource constraints. That brings up a situation where cellphone vendors, if they are doing their jobs right at least, needs to support security fixes across several branches and backport security fixes.

This backporting is actually very difficult to do, because it require a large security department identifying whether the software bugs are security related, affects the various branches, modifying it the source code to fix the issues on these branches that might have different logic. As such, the choice of cellphone vendor needs to include a consideration of their ability to track security upgrades across the phones and properly defined support cycles for the various phones involved. This is why other software products have End of Life statements for when security fixes are no longer issued for a specific version branch.

Of course, these fixes doesn't matter if the users don't update their systems to receive the fixes. For cellphones this is actually one of the better parts; you see a much broader update for this compared to e.g. regular computers. But the part of the cellphone vendors fixing things is sadly lacking, in particular due to backporting to old kernel versions.

Lets move away from the technical for a little bit and go back to the Sonos system mentioned initially. Ultimately consumers wants things to be easy, and they want everything to communicate directly, e.g. using the cellphone to control the music playing in the living room. That is perfectly natural, but in order to accomodate this, the easy solution is to allow direct access between all network nodes. As described in my former blog post Your Weakest Security Link? Your Children, or is it?  this isn't necessarily a good idea, in fact, it is likely a very bad idea.

I started out mentioning Sonos as that is what prompted me to write this article, in frustration after trying to set up this system on a segregated network, completely isolatet, yet it kept requiring internet access for software updates even to allow to play musing through the digital SPDIF cable. This was supposed to have been one of the easier setups possible, connected to TV and a multi-media computer running Gentoo Linux for things like streaming Netflix and HBO. I would never allow a "smart" application to run unrestricted on other applicances, and I very much like to lock it down as much as possible, using it as, guess what - a sound system. However, the constant requests for updates before it can be used means that you open up a channel for data lekage out of your home, for now that means opening up this specifically in the firewall rules whenever an update is necessary to proceed, but of course, an attacker could make use of this and just submit data along with the update request in batch job rather than streaming live.

Devices have bugs, in particular devices that are outside of your control is of worry; and lets face it, reading through the applications requests for access to information when trying to install a new one results in very few apps being permitted on my own cellphone, can you expect others ensuring proper security hygiene with their own devices? Even my own devices like these are considered as non-secured and should not be permitted access to the regular network. This means setting up an isolated network where they only have access to services explicitly granted permission to, but not between eachother so that it can spread malware and monitor use.

We solved this in the setup the blog post started out about by setting up a new WiFi for Appliances that does not have internet access. You might ask "Why not?" and happily plug your "Smart" TV to the regular network, but history has shown that is a bad idea:

Samsung's small print says that its Smart TV's voice recognition system will not only capture your private conversations, but also pass them onto third parties.

And it is not the only device having the capability of doing so. The only way around this is complete network segregation and a security boundry that doesn't allow traffic (neither upstream nor downstream) unless you explicitly want to grant it.

Expecting users to properly configure their networks is however a pipe dream, and here comes an important issue. The less users care about their privacy, either it is allowing recording devices in your living room, or cellphones that snaps photos without you knowing it , you are at increased risk. There is absolutely no doubt that the choices of other users influence your own security, that further require reduced privileges of any network your guests are permitted into or more careful examination of information that you share (nor not) with them given their lack of ability to safeguard it, reducing your trust in them.

I'm worried about the continued trend of lack of privacy and security focus; but rather a focus on rapid functionality development, increased interconnectiveness and complexity of systems, without a focus on security and privacy that ensures the architecture is sustainable.

Stopping this blog post for now, as to ensure the rant doesn't become too long (or that is maybe too late alreday), but leaving it with a quote from this zdnet article:

The Internet of Things is a safety issue, and therefore a business risk;
When you merge the physical and the digital, it's not just about InfoSec any more. People's lives could be at risk.


September 07, 2015
Bernard Cafarelli a.k.a. voyageur (homepage, bugs)
Testing clang 3.7.0 OpenMP support (September 07, 2015, 15:57 UTC)

Soon after llvm 3.7.0 reached the Gentoo tree, Jeremi Piotrowski opened two bugs to fix OpenMP support in our ebuilds. sys-devel/llvm-3.7.0-r1 (a revision bump that also now installs LLVM utilities) now has a post-install message with a short recap, but here is the blog version.

As detailed in upstream release notes, OpenMP in clang is disabled by default (and I kept this default in the Gentoo package), and the runtime is a separate package. So:

  • emerge sys-libs/libomp
  • add “-fopenmp=libomp” in your CFLAGS
  • check with a sample program like:
  • #include <stdio.h>
    #include <omp.h>
    int main() {
    #pragma omp parallel
        printf("Hello from thread %d, nthreads %d\n", omp_get_thread_num(), omp_get_num_threads());
  • You should get multiple lines on STDOUT (default threads count is one per CPU):
  • % clang -fopenmp=libomp test_omp.c -o test_omp
    % OMP_NUM_THREADS=6 ./test_omp
    Hello from thread 4, nthreads 6
    Hello from thread 5, nthreads 6
    Hello from thread 1, nthreads 6
    Hello from thread 3, nthreads 6
    Hello from thread 0, nthreads 6
    Hello from thread 2, nthreads 6

September 06, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Using multiple OpenSSH daemons (September 06, 2015, 14:37 UTC)

I administer a couple of systems which provide interactive access by end users, and for this interactive access I position OpenSSH. However, I also use this for administrative access to the system, and I tend to have harder security requirements for OpenSSH than most users do.

For instance, on one system, end users with a userid + password use the sFTP server for publishing static websites. Other access is prohibited, so I really like this OpenSSH configuration to use chrooted users, internal sftp support, whereas a different OpenSSH is used for administrative access (which is only accessible by myself and some trusted parties).

Running multiple instances

Although I might get a similar result with a single OpenSSH instance, I prefer to have multiple instances for this. The default OpenSSH port is used for the non-administrative access whereas administrative access is on a non-default port. This has a number of advantages...

First of all, the SSH configurations are simple and clean. No complex configurations, and more importantly: easy to manage through configuration management tools like SaltStack, my current favorite orchestration/automation tool.

Different instances also allow for different operational support services. There is different monitoring for end-user SSH access versus administrative SSH access. Also the fail2ban configuration is different for these instances.

I can also easily shut down the non-administrative service while ensuring that administrative access remains operational - something important in case of changes and maintenance.

Dealing with multiple instances and SELinux

Beyond enabling a non-default port for SSH (i.e. by marking it as ssh_port_t as well) there is little additional tuning necessary, but that doesn't mean that there is no additional tuning possible.

For instance, we could leverage MCS' categories to only allow users (and thus the SSH daemon) access to the files assigned only that category (and not the rest) whereas the administrative SSH daemon can access all categories.

On an MLS enabled system we could even use different sensitivity levels, allowing the administrative SSH to access the full scala whereas the user-facing SSH can only access the lowest sensitivity level. But as I don't use MLS myself, I won't go into detail for this.

A third possibility would be to fine-tune the permissions of the SSH daemons. However, that would require different types for the daemon, which requires the daemons to be started through different scripts (so that we first transition to dedicated types) before they execute the SSHd binary (which has the sshd_exec_t type assigned).

Requiring pubkey and password authentication

Recent OpenSSH daemons allow chaining multiple authentication methods before access is granted. This allows the systems to force SSH key authentication first, and then - after succesful authentication - require the password to be passed on as well. Or a second step such as Google Authenticator.

AuthenticationMethods publickey,password
PasswordAuthentication yes

I don't use the Google Authenticator, but the Yubico PAM module to require additional authentication through my U2F dongle (so ssh key, password and u2f key). Don't consider this three-factor authentication: one thing I know (password) and two things I have (U2F and ssh key). It's more that I have a couple of devices with a valid SSH key (laptop, tablet, mobile) which are of course targets for theft.

The chance that both one of those devices is stolen together with the U2F dongle (which I don't keep attached to those devices of course) is somewhat less.

Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
Printers in Linux ... (September 06, 2015, 11:46 UTC)

I was trying to make a Brother printer do some printing. (Optimistic, I know ... but ...)
Out of the box CUPS can't handle it. Since it's CUPS it doesn't do errors, so who knows why. Anyway, the autoconfigured stuff ends up being eh whut ur doin is naught gud.

The config file points at a hostname that is not even within our local network, so I think this won't print locally. Cloud printing? Cups can do!
There's a printer driver from Brother, which is horribly stupid, bad, stupid, and bad. I read through the 'installer' script trying to figure out what it would do, but then I realized that I'm not on an RPM distro so it's futile.

So then I figured "why not postscript?"

And guess what. All the documentation I found was needlessly wrong and broken, when all you have to do is configure an IPP printer, generic postscript, and ... here comes instant good quality colour duplex output from the printer.

I'm confused about many things here - the complexity of Brother's attempt at a driver that doesn't work, CUPS autoconfiguration that sees the printer and then derps out, and all the documentation that doesn't point out that all this weird layercake of workarounds is not needed because it's a standards-compliant postscript printer.
How exhausting.

September 05, 2015
Hanno Böck a.k.a. hanno (homepage, bugs)

At the recent Chaos Communication Camp I held a talk summarizing the problems with TLS interception or Man-in-the-Middle proxies. This was initially motivated by the occurence of Superfish and my own investigations on Privdog, but I learned in the past month that this is a far bigger problem. I was surprised and somewhat shocked to learn that it seems to be almost a default feature of various security products, especially in the so-called "Enterprise" sector. I hope I have contributed to a discussion about the dangers of these devices and software products.

There is a video recording of the talk avaliable and I'm also sharing the slides (also on Slideshare).

I noticed after the talk that I had a mistake on the slides: When describing Filippo's generic attack on Komodia software I said and wrote SNI (Server Name Indication) on the slides. However the feature that is used here is called SAN (Subject Alt Name). SNI is a feature to have different certificates on one IP, SAN is a feature to have different domain names on one certificate, so they're related and I got confused, sorry for that.

I got a noteworthy comment in the discussion after the talk I also would like to share: These TLS interception proxies by design break client certificate authentication. Client certificates are rarely used, however that's unfortunate, because they are a very useful feature of TLS. This is one more reason to avoid any software that is trying to mess with your TLS connections.

September 03, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)


This post has some horribly saddening photos that are difficult to see. Also, as it is political in nature, it also contains my own personal viewpoints. My goal was not to offend, but to offer my thoughts.

If you’ve been watching the news over the past days and weeks, you’ve certainly seen information regarding the thousands of people fleeing Syria (and other nations in the Middle East [such as Afghanistan] and northern Africa [such as Eritrea]). Natives of these countries have been trying to escape due to the vast political and ethical injustices amidst their war-torn homes.

I had read all the numbers regarding the immigrants, and how the sheer amount of people trying to get into the EU (and possibly seek asylum) was crippling transportation systems, and could be damaging to the economy. A quick recap shows that approximately 350,000 people have migrated into the EU since the start of 2015, and that nearly 100,000 of them hailed from Syria (for a full look at the numbers, and an overview of the situation, please see the “Why is EU struggling with migrants and asylum?” article at BBC News). Looking at the numbers and reading about the conflicts and struggles, though, does not really illuminate the tragedy associated with the events. Though I generally don’t like the hype and drama that several media outlets try to foster, I think that in situations like this one, seeing evidence of the pain and suffering can bring the abominable nature to light.

One such story has been about a Syrian toddler named Aylan Kurdi. A photo of him and his older brother, Galip, can be seen below.

3-year-old Aylan Kurdi with his brother Galip

He and his family were trying to flee their hometown of Kobani, Syria and seek asylum in the EU. They, like many others, were trying to get away from the conflict, the war, and the brutality running rampant in their home. Unfortunately for little Aylan, he would never get to see this new life in the EU. Instead, he drowned, and his body was found washed up on the shore near the town of Bodrum, Turkey.

Body of drowned Syrian toddler Aylan Kurdi

He was trying to survive when the boat that his family was on capsized on its way to the Greek island of Kos. It goes without say that his family (including his father, Abdullah, who survived the sinking) is trying to endure the unimaginable despair that comes with the loss of a child. The pain extends much further, though. For instance, the downcast and devastated look on the face of the Turkish paramilitary officer who found the body speaks volumes.

Turkish officer carrying the body of Syrian toddler Aylan Kurdi

What was the reason for the loss of this little one? To briefly cover it—which unfortunately detracts from the severity of it—the people of Syria started anti-government protests back in 2011. The protests turned violent as supporters of President Bashar al-Assad and those who opposed his rule started fearing for their safety. The full-on civil war became drastically worse in August of 2013 when chemical weapons were fired into suburban and rural areas surrounding the capital city of Damascus. The principal causes of the conflict were political and religious in nature (fighting for democracy against a tyrannical ruler, and going against the Islamic State).

So, due to strongly-held political and religious beliefs, thousands of people have lost their lives whilst myriad others have had their dignity stripped from them. Even with involvement from outside nations, the conflict in Syria has not been adequately resolved, and may not be for many years to come. British Member of Parliament, Nadhim Zahawi posted on his Twitter feed a personal apology to Aylan Kurdi:

Twitter post by British MP Nadhim Zahawi about failing Syria

More important than the particular apology were the words that started his tweet, and the acronym with which he closed it. He opens with “We r [sic] nothing without compassion.” When we get so tangled up in our beliefs (political, religious, or otherwise) that we forget to love one another, we have lost sight of our own humanity. He then closes with “RIP,” which, at first glance, doesn’t sound of out the ordinary. However, it struck me because it stands for “rest in peace.” At a mere three years of age, why wasn’t little Aylan allowed to “LIVE in peace?” Matter of fact, why is anyone disallowed the opportunity to “live in peace” instead of “resting in peace” after death?

In my opinion, the particular causes for the rift in Syria are not where we should be focusing our attention. Instead, we need to remember what the true cost is: the loss of human life, livelihood, and personal freedoms. Again in my opinion, we as people should not hold beliefs of any kind so strongly that we forget about the dignity of others. Protesting to uphold our beliefs, struggling to preserve our ways of life, and pushing to uphold personal freedoms are all noble causes, but they should never supplant our fundamental responsibility of treating other people with respect and love. That is my main point. So, I applaud the media outlets for making the Syrian conflict “real” by showing photos that truly represent the devastating effects of war, instead of just spewing numbers that abstract us from the reality of it. However, they still seemed to miss the key point that nothing warrants the loss of human life.

I realise that this post is difficult to read and view, especially because of the horrific images, but the improper treatment of people is a topic about which I feel very strongly. I would like to end the post with something a little more positive regarding the migrants, though, and show an image that indicates it isn’t all terrible. The photo below is of a Bavarian officer who managed to get a young refugee boy to smile as he entered the capital city of Munich.

Bavarian soldier makes a young Syrian refugee boy smile as he enters Munich
Click to enlarge

Please remember that, no matter how passionately you hold your beliefs, nothing is worth hurting another person. Buddha taught that there is an unalterable law that “Hatred does not cease through hatred at any time. Hatred ceases through love.” Love one another, and only then will these types of inhumanities stop.



Hatred does not cease through hatred at any time. Hatred ceases through love. This is an unalterable law. –Buddha

September 02, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Maintaining packages and backporting (September 02, 2015, 18:33 UTC)

A few days ago I committed a small update to policycoreutils, a SELinux related package that provides most of the management utilities for SELinux systems. The fix was to get two patches (which are committed upstream) into the existing release so that our users can benefit from the fixed issues without having to wait for a new release.

Getting the patches

To capture the patches, I used git together with the commit id:

~$ git format-patch -n -1 73b7ff41
~$ git format-patch -n -1 4fbc6623

The two generated patch files contain all information about the commit. Thanks to the epatch support in the eutils.eclass, these patch files are immediately usable within Gentoo's ebuilds.

Updating the ebuilds

The SELinux userspace ebuilds in Gentoo all have live ebuilds available which are immediately usable for releases. The idea with those live ebuilds is that we can simply copy them and commit in order to make a new release.

So, in case of the patch backporting, the necessary patch files are first moved into the files/ subdirectory of the package. Then, the live ebuild is updated to use the new patches:

@@ -88,6 +85,8 @@ src_prepare() {
                epatch "${FILESDIR}/0070-remove-symlink-attempt-fails-with-gentoo-sandbox-approach.patch"
                epatch "${FILESDIR}/0110-build-mcstrans-bug-472912.patch"
                epatch "${FILESDIR}/0120-build-failure-for-mcscolor-for-CONTEXT__CONTAINS.patch"
+               epatch "${FILESDIR}/0130-Only-invoke-RPM-on-RPM-enabled-Linux-distributions-bug-534682.patch"
+               epatch "${FILESDIR}/0140-Set-self.sename-to-sename-after-calling-semanage-bug-557370.patch"

        # rlpkg is more useful than fixfiles

The patches themselves do not apply for the live ebuilds themselves (they are ignored there) as we want the live ebuilds to be as close to the upstream project as possible. But because the ebuilds are immediately usable for releases, we add the necessary information there first.

Next, the new release is created:

~$ cp policycoreutils-9999.ebuild policycoreutils-2.4-r2.ebuild

Testing the changes

The new release is then tested. I have a couple of scripts that I use for automated testing. So first I update these scripts to also try out the functionality that was failing before. On existing systems, these tests should fail:

Running task semanage (Various semanage related operations).
    Executing step "perm_port_on   : Marking portage_t as a permissive domain                              " -> ok
    Executing step "perm_port_off  : Removing permissive mark from portage_t                               " -> ok
    Executing step "selogin_modify : Modifying a SELinux login definition                                  " -> failed

Then, on a test system where the new package has been installed, the same testset is executed (together with all other tests) to validate if the problem is fixed.

Pushing out the new release

Finally, with the fixes in and validated, the new release is pushed out (into ~arch first of course) and the bugs are marked as RESOLVED:TEST-REQUEST. Users can confirm that it works (which would move it to VERIFIED:TEST-REQUEST) or we stabilize it after the regular testing period is over (which moves it to RESOLVED:FIXED or VERIFIED:FIXED).

I do still have to get used to Gentoo using git as its repository now. The workflow to use is documented though. Luckily, because I often get that the git push fails (due to changes to the tree since my last pull). So I need to run git pull --rebase=preserve followed by repoman full and then the push again sufficiently quick after each other).

This simple flow is easy to get used to. Thanks to the existing foundation for package maintenance (such as epatch for patching, live ebuilds that can be immediately used for releases and the ability to just cherry pick patches towards our repository) we can serve updates with just a few minutes of work.

Bernard Cafarelli a.k.a. voyageur (homepage, bugs)
LLVM 3.7.0 released, Gentoo package changes (September 02, 2015, 11:53 UTC)

As announced here, the new LLVM release is now available. If you are interested after reading the release notes (or the clang ones), I just added the corresponding packages in Gentoo, although still masked for some additional testing. mesa-11.0 release candidates compile fine with it though.
2015/09/04 edit: llvm 3.7.0 is now unmasked!

On the Gentoo side, this version bump has some nice changes:

  • We now use the cmake/ninja build system: huge thanks to fellow developer mgorny for his hard work on this rewrite. This is the recommended build system upstream now, and some features will only be available with it
  • LLDB, the debugger  is now available via USE=lldb. This is not a separate package as the build system for llvm and subprojects is quite monolithic. clang is in the same situation: sys-devel/clang package currently does almost nothing, the magic is done in llvm[clang])
  • These new ebuilds include the usual series of fixes, including some recent ones by gienah

On the TODO list, some tests are still failing on 32bit multilib (bug #558604), ocaml doc compilation/installation has a strange bug (see the additional call on docs/ocaml_doc target in the ebuild). And let’s not forget adding other tools, like polly. Another recent suggestion: add llvm’s OpenMP library needed for OpenMP with clang.

Also, if people were still using the dragonegg GCC plugin, note that it is not released anymore upstream. As 3.6.x versions had compilation problems, this package received its last rites.