Gentoo Logo
Gentoo Logo Side
Gentoo Spaceship

. Aaron W. Swenson
. Agostino Sarubbo
. Alec Warner
. Alex Alexander
. Alex Legler
. Alexey Shvetsov
. Alexis Ballier
. Alexys Jacob
. Amadeusz Żołnowski
. Andreas K. Hüttel
. Andreas Proschofsky
. Anthony Basile
. Arun Raghavan
. Bernard Cafarelli
. Bjarke Istrup Pedersen
. Brent Baude
. Brian Harring
. Christian Ruppert
. Chí-Thanh Christopher Nguyễn
. Daniel Gryniewicz
. David Abbott
. Denis Dupeyron
. Detlev Casanova
. Diego E. Pettenò
. Domen Kožar
. Donnie Berkholz
. Doug Goldstein
. Eray Aslan
. Fabio Erculiani
. Gentoo Haskell Herd
. Gentoo Monthly Newsletter
. Gentoo News
. Gilles Dartiguelongue
. Greg KH
. Hanno Böck
. Hans de Graaff
. Ian Whyman
. Ioannis Aslanidis
. Jan Kundrát
. Jason Donenfeld
. Jeffrey Gardner
. Jeremy Olexa
. Joachim Bartosik
. Johannes Huber
. Jonathan Callen
. Jorge Manuel B. S. Vicetto
. Joseph Jezak
. Kenneth Prugh
. Kristian Fiskerstrand
. Lance Albertson
. Liam McLoughlin
. LinuxCrazy Podcasts
. Luca Barbato
. Luis Francisco Araujo
. Mark Loeser
. Markos Chandras
. Mart Raudsepp
. Matt Turner
. Matthew Marlowe
. Matthew Thode
. Matti Bickel
. Michael Palimaka
. Michal Hrusecky
. Michał Górny
. Mike Doty
. Mike Gilbert
. Mike Pagano
. Nathan Zachary
. Ned Ludd
. Nirbheek Chauhan
. Pacho Ramos
. Patrick Kursawe
. Patrick Lauer
. Patrick McLean
. Pavlos Ratis
. Paweł Hajdan, Jr.
. Petteri Räty
. Piotr Jaroszyński
. Rafael Goncalves Martins
. Raúl Porcel
. Remi Cardona
. Richard Freeman
. Robin Johnson
. Ryan Hill
. Sean Amoss
. Sebastian Pipping
. Steev Klimaszewski
. Stratos Psomadakis
. Sune Kloppenborg Jeppesen
. Sven Vermeulen
. Sven Wegener
. Thomas Kahle
. Tiziano Müller
. Tobias Heinlein
. Tobias Klausmann
. Tom Wijsman
. Tomáš Chvátal
. Vikraman Choudhury
. Vlastimil Babka
. Zack Medico

Last updated:
July 03, 2015, 02:06 UTC

Views expressed in the content published here do not necessarily represent the views of Gentoo Linux or the Gentoo Foundation.

Bugs? Comments? Suggestions? Contact us!

Powered by:
Planet Venus

Welcome to Gentoo Universe, an aggregation of weblog articles on all topics written by Gentoo developers. For a more refined aggregation of Gentoo-related topics only, you might be interested in Planet Gentoo.

June 27, 2015
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
LastPass got hacked, I'm still okay with it (June 27, 2015, 13:23 UTC)

So LastPass was compromised and so they report. I'm sure there are plenty of smug geeks out there, happy about users being compromised. I thought that this is the right time to remind people why I'm a LastPass user and will stay a LastPass user even after this.

The first part is a matter of trust in the technology. If I did not trust LastPass enough to not have easy access to the decrypted content, I wouldn't be using it to begin with. Since I do not trust the LastPass operators, even in the case the encrypted vault were compromised (and they say they weren't), I wouldn't be worrying too much.

On the other hand I followed the obvious course of not only changing the master password, and change the important passwords just to be paranoid. This is actually one good side of LastPass — changing the passwords that are really important is very easy as they instrument the browser, so Facebook, Twitter, Amazon, PayPal, … are one click away from a new, strong password.

Once again, the main reason why I suggest tools such as LastPass (and I like LastPass, but that's just preference) is that they are easy to use, and easy to use means people will use them. Making tools that are perfectly secure in theory but very hard to use just means people will not use them, full stop. A client-side certificate is much more secure than a password, but at the same time procuring one and using it properly is non-trivial so in my experience only a handful of services use that — I know of a couple of banks in Italy, and of course StartSSL and similar providers.

The problem with offline services is that, for the most part, don't allow good access while from phones, for instance. So you end up choosing, for things you use often from the phone, memorable passwords. But memorable passwords are usually fairly easy to crack, unless you use known methods and long password — although at least it's not the case, like I read on Arse^H recently, that since we know the md5 hash for "mom", any password with that string anywhere is weakened.

Let's take an example away from the password vaults. In Ireland (and I assume UK simply because the local systems are essentially the same in many aspects), banks have this bollocks idea that is more secure to ask for some of the characters of a password rather than a full password. I think this is a remnant of old bank teller protocols, as I remember reading about that in The Art of Deception (good read, by the way.)

While in theory picking a random part of the password means a phishing attempt would never get the full password, and thus won't be able to access the bank's website unless they are very lucky and get exactly the same three indexes over and over, it is a frustrating experience.

My first bank, AIB, used a five-digits PIN, and then select three digits out of it when I log in, which is not really too difficult to memorize. On the other hand, on their mobile app they decided that the right way to enter the numbers is by using drop-down boxes (sigh.) My current bank, Ulster Bank/RBS, uses a four digits pin, plus a variable length password, which I generated through LastPass as 20 characters, before realizing how bad that is, because it means I now get asked three random digits off the four... and three random characters of the 20.

Let that sink in a moment: they'll ask me for the second, fifth and sixteenth character of a twenty characters randomly generated password. So no auto-fill, no copy-paste, no password management software assisted login. Of course most people here would just not bother and go with a simple password they can remember. Probably made of multiple words of the same length (four letters? five?) so that it becomes easy to count which one is the first character of the fourth word (sixteenth character of the password.) Is it any more secure?

I think I'll write a separate blog post about banks apps and website security mis-practices because it's going to be a long topic and one I want to write down properly so I can forward it to my bank contacts, even though it won't help with anything.

Once again, my opinion is that any time you make security a complicated feature, you're actually worsening the practical security, even if your ideas are supposed to improve the theoretical one. And that includes insisting on the perfect solution for password storage.

June 26, 2015
Mike Pagano a.k.a. mpagano (homepage, bugs)
kdbus in gentoo-sources (June 26, 2015, 23:35 UTC)


Keeping with the theme of ‘Gentoo is about choice” I’ve added the ability for users to include kdbus into their gentoo-sources kernel.  I wanted an easy way for gentoo users to test the patchset while maintaining the default installation of not having it at all.

In order to include the patchset on your gentoo-sources you’ll need the following:

1. A kernel version >= 4.1.0-r1

2. the ‘experimental’ use flag

3. the ‘kdbus’ use flag

I am not a systemd user, but from the ebuild it looks like if you build systemd with the ‘kdbus’ use flag it will use it.

Please send all kdbus bugs upstream by emailing the developers and including in the CC .

Read as much as you can about kdbus before you decided to build it into your kernel.  There have been security concerns mentioned (warranted or not), so following the upstream patch review at would probably be prudent.

When a new version is released, wait a week before opening a bug.  Unless I am on vacation, I will most likely have it included before the week is out. Thanks!

NOTE: This is not some kind of Gentoo endorsement of kdbus.  Nor is it a Mike Pagano endorsement of kdbus.  This is no different then some of the other optional and experimental patches we carry.  I do all the genpatches work which includes the patches, the ebuilds and the bugs therefore since I don’t mind the extra work of keeping this up to date, then I can’t see any reason not to include it as an option.



June 25, 2015
Johannes Huber a.k.a. johu (homepage, bugs)
KDE Plasma 5.3.1 testing (June 25, 2015, 23:15 UTC)

After several month of packaging in kde overlay and almost a month in tree, we have lifted the mask for KDE Plasma 5.3.1 today. If you want to test it out, now some infos how to get it.

For easy transition we provide two new profiles, one for OpenRC and the other for systemd.

root # eselect profile list
[8] default/linux/amd64/13.0/desktop/plasma
[9] default/linux/amd64/13.0/desktop/plasma/systemd

Following example activates the Plasma systemd profile:

root # eselect profile set 9

On stable systems you need to unmask the qt5 use flag:

root # echo "-qt5" >> /etc/portage/profile/use.stable.mask

Any packages affected by the profile change need to be rebuilt:

root # emerge -avuND @world

For stable users, you also need to keyword the required packages. You can let portage handle it with autokeyword feature or just grep the keyword files for KDE Frameworks 5.11 and KDE Plasma 5.3.1 from kde overlay.

Now just install it (this is full Plasma 5, the basic desktop would be kde-plasma/plasma-desktop):

root # emerge -av kde-plasma/plasma-meta

KDM is not supported for Plasma 5 anymore, so if you have installed it kill it with fire. Possible and tested login managers are SDDM and LightDM.

For detailed instructions read the full upgrade guide. Package bugs can be filed to and about the software to

Have fun,
the Gentoo KDE Team

June 23, 2015
Hanno Böck a.k.a. hanno (homepage, bugs)

tl;dr Most servers running a multi-user webhosting setup with Apache HTTPD probably have a security problem. Unless you're using Grsecurity there is no easy fix.

I am part of a small webhosting business that I run as a side project since quite a while. We offer customers user accounts on our servers running Gentoo Linux and webspace with the typical Apache/PHP/MySQL combination. We recently became aware of a security problem regarding Symlinks. I wanted to share this, because I was appalled by the fact that there was no obvious solution.

Apache has an option FollowSymLinks which basically does what it says. If a symlink in a webroot is accessed the webserver will follow it. In a multi-user setup this is a security problem. Here's why: If I know that another user on the same system is running a typical web application - let's say Wordpress - I can create a symlink to his config file (for Wordpress that's wp-config.php). I can't see this file with my own user account. But the webserver can see it, so I can access it with the browser over my own webpage. As I'm usually allowed to disable PHP I'm able to prevent the server from interpreting the file, so I can read the other user's database credentials. The webserver needs to be able to see all files, therefore this works. While PHP and CGI scripts usually run with user's rights (at least if the server is properly configured) the files are still read by the webserver. For this to work I need to guess the path and name of the file I want to read, but that's often trivial. In our case we have default paths in the form /home/[username]/websites/[hostname]/htdocs where webpages are located.

So the obvious solution one might think about is to disable the FollowSymLinks option and forbid users to set it themselves. However symlinks in web applications are pretty common and many will break if you do that. It's not feasible for a common webhosting server.

Apache supports another Option called SymLinksIfOwnerMatch. It's also pretty self-explanatory, it will only follow symlinks if they belong to the same user. That sounds like it solves our problem. However there are two catches: First of all the Apache documentation itself says that "this option should not be considered a security restriction". It is still vulnerable to race conditions.

But even leaving the race condition aside it doesn't really work. Web applications using symlinks will usually try to set FollowSymLinks in their .htaccess file. An example is Drupal which by default comes with such an .htaccess file. If you forbid users to set FollowSymLinks then the option won't be just ignored, the whole webpage won't run and will just return an error 500. What you could do is changing the FollowSymLinks option in the .htaccess manually to SymlinksIfOwnerMatch. While this may be feasible in some cases, if you consider that you have a lot of users you don't want to explain to all of them that in case they want to install some common web application they have to manually edit some file they don't understand. (There's a bug report for Drupal asking to change FollowSymLinks to SymlinksIfOwnerMatch, but it's been ignored since several years.)

So using SymLinksIfOwnerMatch is neither secure nor really feasible. The documentation for Cpanel discusses several possible solutions. The recommended solutions require proprietary modules. None of the proposed fixes work with a plain Apache setup, which I think is a pretty dismal situation. The most common web server has a severe security weakness in a very common situation and no usable solution for it.

The one solution that we chose is a feature of Grsecurity. Grsecurity is a Linux kernel patch that greatly enhances security and we've been very happy with it in the past. There are a lot of reasons to use this patch, I'm often impressed that local root exploits very often don't work on a Grsecurity system.

Grsecurity has an option like SymlinksIfOwnerMatch (CONFIG_GRKERNSEC_SYMLINKOWN) that operates on the kernel level. You can define a certain user group (which in our case is the "apache" group) for which this option will be enabled. For us this was the best solution, as it required very little change.

I haven't checked this, but I'm pretty sure that we were not alone with this problem. I'd guess that a lot of shared web hosting companies are vulnerable to this problem.

Here's the German blog post on our webpage and here's the original blogpost from an administrator at Uberspace (also German) which made us aware of this issue.

June 21, 2015
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)
Perl 5.22 testers needed! (June 21, 2015, 08:14 UTC)

Gentoo users rejoice, for a few days already we have Perl 5.22.0 packaged in the main tree. Since we don't know yet how much stuff will break because of the update, it is masked for now. Which means, we need daring testers (preferably running ~arch systems, stable is also fine but may need more work on your part to get things running) who unmask the new Perl, upgrade, and file bugs if needed!!!
Here's what you need in /etc/portage/package.unmask (and possibly package.accept_keywords) to get started (download); please always use the full block, since partial unmasking will lead to chaos. We're looking forward to your feedback!
# Perl 5.22.0 mask / unmask block

# end of the Perl 5.22.0 mask / unmask block
After the update, first run
emerge --depclean --ask
and afterwards
perl-cleaner --all
perl-cleaner should not need to do anything, ideally. If you have depcleaned first and it still wants to rebuild something, that's a bug. Please file a bug report for the package that is getting rebuilt (but check our wiki page on known Perl 5.22 issues first to avoid duplicates).

June 20, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs)


Von Krautreporter hatte ich parallel über meinen Bruder und durch die “Jung & Naiv” Videos von Tilo Jung erfahren.
Mit meiner Mitgliederschaft als Leser bin ich sehr zufrieden: Viele Extras habe ich nicht genutzt, aber doch hier und da in einen Mitglieder-only-Podcast (“Viertel vor Feierabend”) reingehört und auch die Gelegenheit genutzt, die Redaktion zum Jubiläum vor Ort etwas näher kennenzulernen.

Ich bin auf jeden Fall im zweiten Jahr dabei, nicht als Experiment sondern wegen der Artikel: viel Spannendes dabei. Spannend fand ich persönlich zum Beispiel konkret

und von Theresa Bäuerlein hab ich noch mehrere Bookmarks mit Artikeln, für die ich mir noch Zeit nehmen werde.

Ich hoffe sehr, dass Krautreporter weiter bestehen kann. Vielleicht ist echter Journalismus ohne Werbung ja auch was für dich? Mehr Details gibt’s hier:

June 19, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)


My tech articles—especially Linux ones—are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Recently, I wrote an article about amavisd not running with Postfix, and getting a “Connection refused to″ error message that wasn’t easy to diagnose. Yesterday, I ran into another problem with amavisd refusing to start properly, and I wasn’t readily able to figure out why. By default, amavisd logs to your mail log, which for me is located at /var/log/mail.log, but could be different for you based on your syslogger preferences. The thing is, though, that it will not log start-up errors there. So basically, one is seemingly left in the dark if you start amavisd and then realise it isn’t running immediately thereafter.

I decided to take a look at the init script for amavisd, and saw that there were some non-standard functions in it:

# grep 'extra_commands' /etc/init.d/amavisd
extra_commands="debug debug_sa"

These extra commands map to the following functions:

debug() {
ebegin "Starting ${progname} in debug mode"
"${prog}" debug
eend $?

debug_sa() {
ebegin "Starting ${progname} in debug-sa mode"
"${prog}" debug-sa
eend $?

Though these extra commands may be Gentoo-specific, they are pretty easy to implement on other distributions by directly calling the binary itself. For instance, if you wanted the debug function, it would be the location of the binary with ‘debug’ appended to it. On my system, that would be:

/usr/sbin/amavisd -c $LOCATION_OF_CONFIG_FILE debug

replacing the $LOCATION_OF_CONFIG_FILE with your actual config file location.

When I started amavisd in debug mode, the start-up problem that it was having became readily apparent:

# /etc/init.d/amavisd debug
* Starting amavisd-new in debug mode ...
Jun 18 12:48:21.948 /usr/sbin/amavisd[4327]: logging initialized, log level 5, syslog: amavis.mail
Jun 18 12:48:21.948 /usr/sbin/amavisd[4327]: starting. /usr/sbin/amavisd at amavisd-new-2.10.1 (20141025), Unicode aware, LANG="en_GB.UTF-8"

Jun 18 12:48:22.200 /usr/sbin/amavisd[4327]: Net::Server: 2015/06/18-12:48:22 Amavis (type Net::Server::PreForkSimple) starting! pid(4327)
Jun 18 12:48:22.200 /usr/sbin/amavisd[4327]: (!)Net::Server: 2015/06/18-12:48:22 Unresolveable host [::1]:10024 - could not load IO::Socket::INET6: Can't locate in @INC (you may need to install the Socket6 module) (@INC contains: lib /etc/perl /usr/local/lib64/perl5/5.20.2/x86_64-linux /usr/local/lib64/perl5/5.20.2 /usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux /usr/lib64/perl5/vendor_perl/5.20.2 /usr/local/lib64/perl5 /usr/lib64/perl5/vendor_perl/5.20.1/x86_64-linux /usr/lib64/perl5/vendor_perl/5.20.1 /usr/lib64/perl5/vendor_perl /usr/lib64/perl5/5.20.2/x86_64-linux /usr/lib64/perl5/5.20.2) at /usr/lib64/perl5/vendor_perl/5.20.1/Net/Server/ line 122.\n\n at line 82 in file /usr/lib64/perl5/vendor_perl/5.20.1/Net/Server/
Jun 18 12:48:22.200 /usr/sbin/amavisd[4327]: Net::Server: 2015/06/18-12:48:22 Server closing!

In that code block, the actual error (in bold text) indicates that it couldn’t find the Perl module IO:Socket::INET6. This problem was easily fixed in Gentoo with emerge -av dev-perl/IO-Socket-INET6, but could be rectified by installing the module from your distribution’s repositories, or by using CPAN. In my case, it was caused by my recent compilation and installation of a new kernel that, this time, included IPV6 support.

The point of my post, however, wasn’t about my particular problem with amavisd starting, but rather how one can debug start-up problems with the daemon. Hopefully, if you run into woes with amavisd logging, these debug options will help you track down the problem.


June 14, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs)
European Xiangqi stickers (Chinese chess) (June 14, 2015, 21:58 UTC)


For quite a while I have been looking for someone to design european piece graphics for Xiangqi (Chinese chess) without luck. Early this March I was pointed to Jasmin Scharrer fortunately!

The pieces she designed allowed making sheets of stickers to help kickstart the learning experience of people new to the game, facing Chinese-only gaming hardware: You put stickers on the backside of the pieces and turn them over whenever you are ready to: only some and then some more, all at once, your choice.

This is the stickers on real hardware:

The full sticker sheet takes an A4 page and looks like this:

To print your own, you should use the vector version for quality.

Since we picked CC-BY-4.0 for a license, you could even sell your prints as long as proper attribution to

Jasmin Scharrer (original artwork,,
Sebastian Pipping (post-processing,

for the initial source is done.

Using the same piece graphics, xiangqi-setup also has a piece theme “euro_xiangqi_js” now. With that, you can render European setup images like this one:

$ xiangqi-setup --pieces euro_xiangqi_js --width-cm 7 doc/demo.wxf \

If you need help with any of this or for feedback in general, please feel free to contact me at Best, Sebastian

June 13, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Where does CIL play in the SELinux system? (June 13, 2015, 21:12 UTC)

SELinux policy developers already have a number of file formats to work with. Currently, policy code is written in a set of three files:

  • The .te file contains the SELinux policy code (type enforcement rules)
  • The .if file contains functions which turn a set of arguments into blocks of SELinux policy code (interfaces). These functions are called by other interface files or type enforcement files
  • The .fc file contains mappings of file path expressions towards labels (file contexts)

These files are compiled into loadable modules (or a base module) which are then transformed to an active policy. But this is not a single-step approach.

Transforming policy code into policy file

For the Linux kernel SELinux subsystem, only a single file matters – the policy.## file (for instance policy.29). The suffix denotes the binary format used as higher numbers mean that additional SELinux features are supported which require different binary formats for the SELinux code in the Linux kernel.

With the 2.4 userspace, the transformation of the initial files as mentioned above towards a policy file is done as follows:

When a developer builds a policy module, first checkmodule is used to build a .mod intermediate file. This file contains the type enforcement rules with the expanded rules of the various interface files. Next, semodule_package is called which transforms this intermediate file, together with the file context file, into a .pp file.

This .pp file is, in the 2.4 userspace, called a “high level language” file. There is little high-level about it, but the idea is that such high-level language files are then transformed into .cil files (CIL stands for Common Intermediate Language). If at any moment other frameworks come around, they could create high-level languages themselves and provide a transformation engine to convert these HLL files into CIL files.

For the current .pp files, this transformation is supported through the /usr/libexec/selinux/hll/pp binary which, given a .pp file, outputs CIL code.

Finally, all CIL files (together) are compiled into a binary policy.29 file. All the steps coming from a .pp file towards the final binary file are handled by the semodule command. For instance, if an administrator loads an additional .pp file, its (generated) CIL code is added to the other active CIL code and together, a new policy binary file is created.

Adding some CIL code

The SELinux userspace development repository contains a secilc command which can compile CIL code into a binary policy file. As such, it can perform the (very) last step of the file conversions above. However, it is not integrated in the sense that, if additional code is added, the administrator can “play” with it as he would with SELinux policy modules.

Still, that shouldn’t prohibit us from playing around with it to experiment with the CIL language construct. Consider the following CIL SELinux policy code:

; Declare a test_port_t type
(type test_port_t)
; Assign the type to the object_r role
(roletype object_r test_port_t)

; Assign the right set of attributes to the port
(typeattributeset defined_port_type test_port_t)
(typeattributeset port_type test_port_t)

; Declare tcp:1440 as test_port_t
(portcon tcp 1440 (system_u object_r test_port_t ((s0) (s0))))

The code declares a port type (test_port_t) and uses it for the TCP port 1440.

In order to use this code, we have to build a policy file which includes all currently active CIL code, together with the test code:

~$ secilc -c 29 /var/lib/selinux/mcs/active/modules/400/*/cil testport.cil

The result is a policy.29 (the command forces version 29 as the current Linux kernel used on this system does not support version 30) file, which can now be copied to /etc/selinux/mcs/policy. Then, after having copied the file, load the new policy file using load_policy.

And lo and behold, the port type is now available:

~# semanage port -l | grep 1440
test_port_t           tcp      1440

To verify that it really is available and not just parsed by the userspace, let’s connect to it and hope for a nice denial message:

~$ ssh -p 1440 localhost
ssh: connect to host localhost port 1440: Permission denied

~$ sudo ausearch -ts recent
time->Thu Jun 11 19:35:45 2015
type=PROCTITLE msg=audit(1434044145.829:296): proctitle=737368002D700031343430006C6F63616C686F7374
type=SOCKADDR msg=audit(1434044145.829:296): saddr=0A0005A0000000000000000000000000000000000000000100000000
type=SYSCALL msg=audit(1434044145.829:296): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=6d4d1ce050 a2=1c a3=0 items=0 ppid=2005 pid=18045 auid=1001 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=pts0 ses=1 comm="ssh" exe="/usr/bin/ssh" subj=staff_u:staff_r:ssh_t:s0 key=(null)
type=AVC msg=audit(1434044145.829:296): avc:  denied  { name_connect } for  pid=18045 comm="ssh" dest=1440 scontext=staff_u:staff_r:ssh_t:s0 tcontext=system_u:object_r:test_port_t:s0 tclass=tcp_socket permissive=0

Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Artificial regions, real risks (June 13, 2015, 14:20 UTC)

In my last encounter with what I call "artificial regions" that I talked about, I was complaining about the problems with ping-ponging between US and Italy, and then moving to Dublin. Those first-world problems are now mostly (but not fully) solved and not really common, so I wouldn't call them "real" for most people.

What I have ignored in that series of posts was, though, the region-locking applied by Big Content providers, particularly in regards to movies, TV series, and so on. This was because it's a problem that is way too obvious already, and there isn't much that one can add to it at this point, it has been written about, illustrated and argued for years by now.

The reason why I'm going back to this now, though, is that there has recently been news of yet another scam, at the damage of the final consumers, connected to a common way to work around artificial region limitations. But first, let me point out the obvious first step: in this post I'm not talking about the out-and-out piracy option of downloading content straight from The Pirate Bay or anything along those lines. I'm instead going to focus on those people who either pay for a service, or wants to pay for content, but are blocked by the artificial region set up for content.

I'll use as my first example Comixology of which I'm a customer, because I like comics but I travel too much to bring them with me physically, and more importantly would just increase the amount of things I'd have to move with me if I decide to change place. Unlike many other content providers, Comixology uses multiple regional segregation approaches: your payment card billing address tells you which website you can use, which actually only changes how much you're paying; the IP you're coming from tells you which content you can buy. Luckily, they still let you access paid content even if your IP no longer match the one you can buy it from.

This is not really well documented, by the way. Some time ago they posted on their G+ page that they opened a deal with a manga distributor so that more content was available; I took the chance to buy a bunch of Bleach issues (but not all of them) as I have not finished watching the anime due to other issues in the past, and I wanted to catch up on my terms. But a few weeks later when I wanted to buy more because I finished the stash… I couldn't. I thought they broke off the deal, since there was no reference to it on the website or app, so I gave up… until they posted a sale, and I saw that they did list Bleach, but the link brought me to a "Page not Found" entry.

Turns out that they admitted on Twitter that due to the way the rights for the content go, they are not allowed to sell manga outside of the States, and even though they do validate my billing address (I use my American debit card there) they seem to ignore it and only care on where they think they are physically located at the moment. Which is kinda strange, given that it means you can buy manga from them if you've got an European account and just so happens to travel to the United States.

Admittedly, this is by far not something that only happens with this website. In particular, Google Play Movies used to behave this way, where you could buy content while abroad, but you would be stopped from downloading it (but if you had it downloaded, you could still play.) This was unlike Apple, that always tied its availability and behaviour on the country your iTunes account was tied to, verified by billing address of the connected payment card — or alternatively faked and paid with country-specific iTunes gift cards.

One "easy" way to work around this is to use VPN services to work around IP-based geographical restrictions. The original idea of a VPN is to connect multiple LANs, or a remote client to a LAN, in a secure way. While there is some truth about the security of this, lots of it is actually vapourware, due to so many technical hurdles of actually securing a LAN, that the current trend is to not rely on VPNs at all. But then again, VPNs allow you to change where your Internet egress is, which is handy.

A trustworthy VPN is actually a very useful tool, especially if what you're afraid of is sniffers of your traffic on public WiFi and similar, because now you're actually only talking with a single (or limited set of) points-of-presence (POP) with an encrypted protocol. The keyword here is trustworthy, as now instead of worrying of what people in your proximity could do with your non-encrypted traffic, you have to worry what the people who manage the VPN, or in proximity to the VPN provider, would do with that information.

Even more important, since VPNs are generally authenticated, an attacker that can control or infiltrate your VPN provider can easily tie together all your traffic, no matter where you're connecting from. This is possibly the sole thing for which Tor is a better option, as there isn't a single one manager for the VPN — although recent discussions may show that even Tor is not as safe from following a single user as some people kept promising or boasting.

These VPN services end up being advertised as either privacy tools, for the reasons just noted, or as tools against censorship. In the latter case the usual scare is the Great Firewall of China, without considering that there are very few websites that suffer what I'd define "censorship" on a country level — The Pirate Bay does not count, as much as I think it's silly and counterproductive to ban access to it, censorship is a word I'd reserve for much more venomous behaviour. Region-locking, on the other hand, as I've shown is pretty heavy, but just saying out loud that you work around region-locking is not really good for business, as it may well be against all terms of service that people say they accepted.

Here comes the bombshell for most people: yes, VPN services are (for the most part) not managed by angels who want all information in the world to be free. Such "heroes" are mostly created by popular culture and are much rarer than you would think. Painting them as such would be like painting MegaUpload and Kim Dotcom as generous saviours of humanity — which admittedly I've seen too many people, especially in the Free Software and privacy-conscious movements, doing.

VPN services are, for many people, quite profitable. Datacenter bandwidth is getting cheaper and cheaper, while end-user speeds are either capped, or easily capped by the VPN itself. If you make people pay for the service, it's not going to take that many users to pay for the bandwidth, and then start making profits. And many people are happy to pay for the service, either because it's still cheaper than accepting the geographical restrictions or because they go for the privacy candy.

On the other hand there are free VPN services out there, so what about them? I've been surprised before by self-defined privacy advocates suggesting to the masses to use free VPN services, while at the same time avoiding Dropbox, Microsoft and other offerings with the catchphrase «If you're not paying for it, you're the product.» Turns out for free VPN providers, you most definitely are a product.

To use the neologism I so much hate, it's always interesting to figure out how these providers monetize you, and that's not always easy because it may as well be completely passive: they could be siphoning data the same way I described in my previous post, and then use that to target you for more or less legal or ethical interests. Or they could be injecting referral codes when you browse websites with affiliate programs such as Amazon (similarly to how some Chrome extensions used to work, with the difference of being done at the router level) — this is, by the way, one extremely good reason to use HTTPS everywhere, as you can't do that kind of manipulation on protected pages without fiddling with the certificate.

Or, as it became apparent recently, they may be playing with your egress so that sure, you are now going to the Internet through some random US person's address, but at the same time, your address is being used for… something else. Which may be streaming a different version of Netflix – e.g.: the Big Bang Theory is available on French Netflix, but not in the US one – or they may be selling stolen credit card data, or browse for child porn, or whatever else they care to do.

What's the bottom line here? Well, it seems obvious that the current regime of rights that imposes region-locking of content is not only unlikely to be helping the content producers (rather than distributors) much – The Pirate Bay content never was region-locked – but it's also causing harm to people who would otherwise be happy to pay to be able to access it!

I'm not advocating for removing copyright, or that content should be free to all – I may prefer such a situation, but I don't think it's realistic – but I would pretty much like for these people to wake up and realize that if I'm ready to give them money, it would be a good thing for them to accept it without putting me at risk more than if I were not to give them money and just pirate the content.

And now go, and uninstall Hola, for universe's sake!

June 12, 2015

A while back I came across a fairly good article on the need for consciousness regarding children and computer. However I miss a few paragraphs (or even chapters) regarding necessary steps (even without children).

None of this is of course the childrens’ fault, however it does provide a good opportunity to discuss computer security hygiene, and with it the lack of knowledge and focus on security and privacy in current and new generations.

To start off, the article that inspired this post: Your Weakest Security Link? Your Children – WSJ

What do you do when the biggest threat to your cybersecurity lives under your own roof?

It’s a fact of life online: a network is only as strong as its weakest link. For many people, that weakest link is their children. They inadvertently download viruses. They work around security to visit sites their parents don’t want them to. They run up huge bills using their parents’ one-click ordering.

Segregated WiFi networks
My largest concern with the article is that it does not mention the need for a wifi guest network. You should never, ever, allow external devices outside your control, or lower security devices like tablets and smartphones directly onto your LAN, these devices simply doesn’t need it and should be in a separate security zone on the firewall, if you need access to local resources, use OpenVPN to set up a connection from wifi zone to lan zone, or for certain other elements open up specific ports/applications to pass through.

My usual setup involves flashing a router/accesspoint with dd-wrt (a linux based firmware). This allows setting up guest networks / virtual wireless interfaces (although not all devices will allow anything but public / WEP encryption, i.e. no WPA on the virtual interfaces, as these are by definition low security zones, this is however fine). The most important thing is that the guest network is

  • AP isolated, i.e does not allow traffic between individual devices on the network
  • only allows traffic through NAT to wan (internet), i.e. not LAN devices
  • QoS is enabled as to not interfere with your normal use

The guest network should further be restrictive in what traffic it allows, normally your visitors only require HTTP(s), POP3s, IMAPs, SMTPs, SSH (people shouldn’t use non-encrypted transport channels for reading mail so the non-secured alternatives are blocked), and maybe a few other protocols, so the firewall should ensure to be restrictive as a starting point.

A proper network setup remove the primary issues discussed with children’s devices being connected to network, its not any better allowing other unsecured devices onto it, so why don’t you do it properly in the first place?

In fact, with todays growth of the Internet of things, you will also want a dedicated device-network that is internal (no AP-isolation, but doesn’t allow traffic to lan or wan) for TV communication with speakers etc. I simply don’t get people plugging this directly into the regular network. That only leads to situations such as Samsung voice-recording / surveillance everything you say.

Disk encryption
Disk encryption is necessary in order to avoid passing through the security schemes using known passwords mentioned in the article. I remember in the good old days when NTFS first got here, and how easy it was to brute-force SAN files. Without disk encryption, separating profiles (e.g. in windows 7) won’t help you as anyone can boot a live OS (Linux ftw) giving access to all files, including configuration files. But quite frankly, multi-user systems are difficult to secure, hardware today is cheap, you shouldn’t let your kids use your own hardware in the first place, but set up own systems for them.

Seven proxies
Be restrictive in what kind of data you allow to flow through your network. Use Proxies, proxies, and even more proxies, filter all network access appropriately through firewalls, and more importantly, make frequent use of jump servers / gateways.

If you are worried about certain types of virus intrusion, also get a firewall device supporting intrusion detection/prevention systems and an appropriate license to keep this updated. If you are concerned about your childrens internet use, all web traffic should also be forced through a proxy (such as squid) on the firewall level.

Separate core devices of the network and don’t allow traffic into it even on the regular LAN without going through a jump server / gateway, e.g. using OpenVPN. I actually use this myself, for instance even though I connect my laptop to the regular wifi network, I won’t get access to my Virtual Machines without being logged into the VPN. Only certain specified services are exposed by default.

Cloud services
Cloud services are great, I mean, truly, with the number of devices we have today, how else can we keep data available for our own use? The one caveat, you really, truly, REALLY, MUST run the cloud service yourself. ownCloud is a great application, that have multiple clients across operating systems and platforms for this use. File storage, calendar, contact information and a good news reader. Not much more to ask for, and if there is, there is likely already an app for it.

Don’t ever allow friends or family to use iCloud, DropBox, google drive or whatnot. Of course, promote using these services to your enemies, you might want to get access to their data some day.

Nirbheek Chauhan a.k.a. nirbheek (homepage, bugs)

आो मीलो (clap clap clap)
शीलम शालो (clap clap clap)
कच्चा धागा (clap clap clap)
रेस लगा लो (clap clap clap)

(जलदी से)

आो मीलो शीलम शालो कच्चा धागा रेस लगा लो

दस पत्ते तोड़े
एक पत्ता कच्चा
हिरन का बच्चा
हिरन गया पानी में
पकड़ा उस्की नानी ने
नानी गयी लंडन
वहां से लाइ कंगन

कंगन गया टूट (clap)
नानी गयी रूठ (clap)

(और भी तेज़ी से)

नानी को मनाएंगे
रस मालाइ खाएंगे
रस मालाइ अच्छी
हमने खाइ मच्छी
मच्छी में निकला कांटा
मम्मी ने मारा चांटा
चांटा लगा ज़ोर से
हमने खाए समोसे
समोसे बढे अच्छे

नानाजी नमसते!

June 11, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)
The Parker Fund (June 11, 2015, 02:48 UTC)

Hello all,

I don’t usually post very personal things here on The Z-Issue, but rather use it for 1) posting information that may be of interest to others (e.g. solving technical problems [especially with regard to the world of Linux Engineering]), 2) mentioning articles that I found helpful, amusing, or otherwise interesting, or 3) just babbling about who knows what. This article, though, is about something that is very special to me, and one that is intimately and inextricably tied to who I am as a person.

For those of you who have read the about me page, you know that I have devoted the majority of my academic life to studies of Child and Adolescent Development / Psychology and Education. Understanding and helping children work through their difficult times is one of my biggest passions in this life.

Over the past decade or so, I’ve given a lot of thought to having children of my own and have decided that I would rather adopt than have kids who are biologically mine. My rationale is multifaceted, but primarily, I just think that there are enough children already here that could use loving homes, and I would like to offer that to one of them instead of bringing another one into the world. So, for several years now, I’ve been trying to put any little bit of extra money that I have into savings for adopting a child. In the past year, I have started to pursue adoption more actively from a lot of angles (discussing it with attorneys, researching adoption agencies and policies, et cetera). I have also tried to squirrel away even more money by doing side jobs and such. Though I’ve been relatively successful at saving money for the adoption, I have a very long way to go based on the estimate that I have been given (approximately $85,000 USD). At the time of this writing, I have saved a mere ~17% of the funds that I will need in order to get started on the process.

Why bring up any of these things about my personal life and goals? Simply put, the idea of crowdfunding is enticing in this situation. I post my articles hoping that they will help people who find themselves in similar situations (as mentioned, particularly relating to Linux, but also to travels, and so on) as the ones in which I have found myself over the years. And, thankfully, according to the analytics for the site and comments on various posts, people are finding the articles helpful. In particular, a couple of my Linux articles have been viewed tens of thousands of times on certain days. I’ll cut to the chase: if any of my articles have been helpful to you, please consider donating ANY amount that you comfortably can to The Parker Fund–helping me reach my goal of being able to adopt. Absolutely every dollar helps, so any donation will be greatly appreciated!

Why “The Parker Fund?” As I’ve said, I have more actively started to pursue the adoption within the past year or so. I’ve pondered many different aspects of the process, and have decided that I would like to adopt a boy first. I have also chosen the name Parker, and so to keep the goal alive and real in my mind, I refer to him by name, though he may not have even been born yet. Again, to keep my motivation up through setback after setback, I have started to decorate a room for him:

Parker's room
Click to enlarge

These types of little things help keep me focused on my goal, even if they seem a little premature.

So, in effort to expedite the start of the adoption process, if you can spare any funds and would like to help me, you may contribute to The Parker Fund by using the PayPal “Donate” button in the top widget on the right-hand side of the page. As I said, every dollar helps and will be greatly appreciated! Even if you can’t or don’t want to contribute, thank you for reading the Z-Issue and taking time out of your day to read about a personal aspiration that is very near and dear to my heart.


June 10, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Live SELinux userspace ebuilds (June 10, 2015, 18:07 UTC)

In between courses, I pushed out live ebuilds for the SELinux userspace applications: libselinux, policycoreutils, libsemanage, libsepol, sepolgen, checkpolicy and secilc. These live ebuilds (with Gentoo version 9999) pull in the current development code of the SELinux userspace so that developers and contributors can already work with in-progress code developments as well as see how they work on a Gentoo platform.

That being said, I do not recommend using the live ebuilds for anyone else except developers and contributors in development zones (definitely not on production). One of the reasons is that the ebuilds do not apply Gentoo-specific patches to the ebuilds. I would also like to remove the Gentoo-specific manipulations that we do, such as small Makefile adjustments, but let’s start with just ignoring the Gentoo patches.

Dropping the patches makes sure that we track upstream libraries and userspace closely, and allows developers to try and send out patches to the SELinux project to fix Gentoo related build problems. But as not all packages can be deployed successfully on a Gentoo system some patches need to be applied anyway. For this, users can drop the necessary patches inside /etc/portage/patches as all userspace ebuilds use the epatch_user method.

Finally, observant users will notice that “secilc” is also provided. This is a new package, which is probably going to have an official release with a new userspace release. It allows for building CIL-based SELinux policy code, and was one of the drivers for me to create the live ebuilds as I’m experimenting with the CIL constructions. So expect more on that later.

June 07, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs) relaunched (June 07, 2015, 23:02 UTC)


Two months ago, the website redesign was announced. For a few hours now, is following. The similarities in look are not by coincidence: Both design were done by Alex Legler.

The new website is based on Jekyll rather than GuideXML previously. Any bugs you find on the website can be reported on GitHub where the website sources are hosted.

I would like to take the opportunity to thank both Manitu and SysEleven for providing the hardware running (and other Gentoo e.V. services) free of charge. Very kind!

Best, Sebastian

June 06, 2015
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
LibreSSL, OpenSSL, collisions and problems (June 06, 2015, 12:33 UTC)

Some time ago, on the gentoo-dev mailing list, there has been an interesting thread on the state of LibreSSL in Gentoo. In particular I repeated some of my previous concerns about ABI and API compatibility, especially when trying to keep both libraries on the same system.

While I hope that the problems I pointed out are well clear to the LibreSSL developers, I thought reiterating them again clearly in a blog post would give them a wider reach and thus hope that they can be addressed. Please feel free to reshare in response to people hand waving the idea that LibreSSL can be either a drop-in, or stand-aside replacement for OpenSSL.

Last year, when I first blogged about LibreSSL, I had to write a further clarification as my post was used to imply that you could just replace the OpenSSL binaries with LibreSSL and be done with it. This is not the case and I won't even go back there. What I'm concerned about this time is whether you can install the two in the same system, and somehow decide which one you want to use on a per-package basis.

Let's start with the first question: why would you want to do that? Everybody at this point knows that LibreSSL was forked from the OpenSSL code and started removing code that has been needed unnecessary or even dangerous – a very positive thing, given the amount of compatibility kludges around OpenSSL! – and as such it was a subset of the same interface as its parent, thus there would be no reason to wanting the two libraries on the same system.

But then again, LibreSSL never meant to be considered a drop-in replacement, so they haven't cared as much for the evolution of OpenSSL, and just proceeded in their own direction; said direction included building a new library, libtls, that implements higher-level abstractions of TLS protocol. This vaguely matches the way NSS (the Netscape-now-Mozilla TLS library) is designed, and I think it makes sense: it reduces the amount of repetition that needs to be coded in multiple parts of the software stack to implement HTTPS for instance, reducing the chance of one of them making a stupid mistake.

Unfortunately, this library was originally tied firmly to LibreSSL and there was no way for it to be usable with OpenSSL — I think this has changed recently as a "portable" build of libtls should be available. Ironically, this wouldn't have been a problem at all if it wasn't that LibreSSL is not a superset of OpenSSL, as this is where the core of the issue lies.

By far, this is not the first time a problem like this happens in Open Source software communities: different people will want to implement the same concept in different ways. I like to describe this as software biodiversity and I find it generally a good thing. Having more people looking at the same concept from different angles can improve things substantially, especially in regard to finding safe implementations of network protocols.

But there is a problem when you apply parallel evolution to software: if you fork a project and then evolve it on your own agenda, but keep the same library names and a mostly compatible (thus conflicting) API/ABI, you're going to make people suffer, whether they are developers, consumers, packagers or users.

LibreSSL, libav, Ghostscript, ... there are plenty of examples. Since the features of the projects, their API and most definitely their ABIs are not the same, when you're building a project on top of any of these (or their originators), you'll end up at some point making a conscious decision on which one you want to rely on. Sometimes you can do that based only on your technical needs, but in most cases you end up with a compromise based on technical needs, licensing concerns and availability in the ecosystem.

These projects didn't change the name of their libraries, that way they can be used as drop-rebuild replacement for consumers that keep to the greatest common divisor of the interface, but that also means you can't easily install two of them in the same system. And since most distributions, with the exception of Gentoo, would not really provide the users with choice of multiple implementations, you end up with either a fractured ecosystem, or one that is very much non-diverse.

So if all distributions decide to standardize on one implementation, that's what the developers will write for. And this is why OpenSSL will likely to stay the standard for a long while still. Of course in this case it's not as bad as the situation with libav/ffmpeg, as the base featureset is going to be more or less the same, and the APIs that have been dropped up to now, such as the entropy-gathering daemon interface, have been considered A Bad Idea™ for a while, so there are not going to be OpenSSL-only source projects in the future.

What becomes an issue here is that software is built against OpenSSL right now, and you can't really change this easily. I've been told before that this is not true, because OpenBSD switched, but there is a huge difference between all of the BSDs and your usual Linux distributions: the former have much more control on what they have to support.

In particular, the whole base system is released in a single scoop, and it generally includes all the binary packages you can possibly install. Very few third party software providers release binary packages for OpenBSD, and not many more do for NetBSD or FreeBSD. So as long as you either use the binaries provided by those projects or those built by you on the same system, switching the provider is fairly easy.

When you have to support third-party binaries, then you have a big problem, because a given binary may be built against one provider, but depend on a library that depends on the other. So unless you have full control of your system, with no binary packages at all, you're going to have to provide the most likely provider — which right now is OpenSSL, for good or bad.

Gentoo Linux is, once again, in a more favourable position than many others. As long as you have a full source stack, you can easily choose your provider without considering its popularity. I have built similar stacks before, and my servers deploy stacks similarly, although I have not tried using LibreSSL for any of them yet. But on the desktop it might be trickier, especially if you want to do things like playing Steam games.

But here's the harsh reality, even if you were to install the libraries in different directories, and you would provide a USE flag to choose between the two, it is not going to be easy to apply the right constraints between final executables and libraries all the way into the tree.

I'm not sure if I have an answer to balance the ability to just make the old software use the new library and the side-installation. I'm scared that a "solution" that can be found to solve this problem is bundling and you can probably figure out that doing so for software like OpenSSL or LibreSSL is a terrible idea, given how fast you should update in response to a security vulnerability.

June 05, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)


This article is one of the most-viewed on The Z-Issue, and is sometimes read thousands of times per day. If it has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

So, for several weeks now, I’ve been struggling with disk naming and UDEV as they relate to RHEL or CentOS Kickstart automation via PXE. Basically, what used to be a relatively straight-forward process of setting up partitions based on disk name (e.g. /dev/sda for the primary disk, /dev/sdb for the secondary disk, and so on) has become a little more complicated due to the order in which the kernel may identify disks with UDEV implementations. The originally-used naming conventions of /dev/sdX may no longer be consistent across reboots or across machines. How this problem came to my attention was when I was Kickstarting a bunch of hosts, and realised that several of them were indicating that there wasn’t enough space on the drives that I had referenced. After doing some digging, I realised that some servers were identifying KVM-based media (like via a Cisco UCS CIMC interface), or other USB media as primary disks instead of the actual primary SCSI-based disks (SATA or SAS drives).

Though I agree that identifying disks by their path, their UUID, or an otherwise more permanent name is preferred to the ambiguous /dev/sdX naming scheme, it did cause a problem, and one that didn’t have a readily-identifiable workaround for my situation. My first thought was to use UUID, or gather the disk ID from /dev/disk/by-id/. However, that wouldn’t work because it wouldn’t be the same across all servers, so automatic installations via PXE wouldn’t be feasible. Next, I looked at referencing disks by their PCI location, which can be found within /dev/disk/by-path/. Unfortunately, I found that the PCI location might vary between servers as well. For example, I found:

Server 1:
/dev/disk/by-path/pci-0000:82:00.0-scsi-0:2:0:0 –> primary SCSI disk
/dev/disk/by-path/pci-0000:82:00.0-scsi-0:2:0:0 –> secondary SCSI disk

Server 2:
/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:2:0:0 –> primary SCSI disk
/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:2:0:0 –> secondary SCSI disk

That being said, I did notice one commonality between the servers that I used for testing. The primary and secondary disks (which, by the way, are RAID arrays attached to the same controller) all ended with ‘scsi-0:2:0:0 ‘ for the primary disk, and ‘scsi-0:2:1:0 ‘ for the secondary disk. Thinking that I could possibly just specify using a wildcard, I tried:

part /boot --fstype ext2 --size=100 --ondisk=disk/by-path/*scsi-0:2:0:0

but alas, that caused Anaconda to error out stating that the specified drive did not exist. At this point, I thought that all hope was lost in terms of automation. Then, however, I had a flash of genius (they don’t happen all that often). I could probably gather the full path ID in a pre-installation script, and then get the FULL path ID into the Kickstart configuration file with an include. Here are the code snippets that I wrote:

%pre --interpreter=/bin/bash
## Get the full by-path ID of the primary and secondary disks
osdisk=$(ls -lh /dev/disk/by-path/ | grep 'scsi-0:2:0:0 ' | awk '{print $9}')
datadisk=$(ls -lh /dev/disk/by-path/ | grep 'scsi-0:2:1:0 ' | awk '{print $9}')

## Create a temporary file with the partition scheme to be included in the KS config
echo "# Partitioning scheme" > /tmp/partition_layout
echo "clearpart --all" >> /tmp/partition_layout
echo "zerombr" >> /tmp/partition_layout
echo "part /boot --fstype ext2 --size=100 --ondisk=disk/by-path/$osdisk" >> /tmp/partition_layout
echo "part swap --recommended --ondisk=disk/by-path/$osdisk" >> /tmp/partition_layout
echo "part pv.01 --size=100 --grow --ondisk=disk/by-path/$osdisk" >> /tmp/partition_layout
echo "part pv.02 --size=100 --grow --ondisk=disk/by-path/$datadisk" >> /tmp/partition_layout
echo "volgroup vg_os pv.01" >> /tmp/partition_layout
echo "volgroup vg_data pv.02" >> /tmp/partition_layout
echo "logvol / --vgname=vg_os --size=100 --name=lv_os --fstype=ext4 --grow" >> /tmp/partition_layout
echo "logvol /data --vgname=vg_data --size=100 --name=lv_data --fstype=ext4 --grow" >> /tmp/partition_layout
echo "# Bootloader location" >> /tmp/partition_layout
echo "bootloader --location=mbr --driveorder=/dev/disk/by-path/$osdisk,/dev/disk/by-path/$datadisk --append=\"rhgb\"" >> /tmp/partition_layout

Some notes about the pre-installation script would be: 1) yes, I know that it is a bit clumsy, but it functions despite the lack of elegance; 2) in lines 3 and 4, make sure that the grep statements include a final space. The final space (e.g. grep 'scsi-0:2:0:0 ' instead of just grep 'scsi-0:2:0:0') is important because the machine may have partitions already set up, and in that case, those partitions would be identified as '*scsi-0:2:0:0-part#' where # is the partition number.

So, this pre-installation script generates a file called /tmp/partition_layout that essentially has a normal partition scheme for a Kickstart configuration file, but references the primary and secondary SCSI disks (again, as RAID arrays attached to the same controller) by their full ‘by-path’ IDs. Then, the key is to include that file within the Kickstart configuration via:

# Partitioning scheme information gathered from pre-install script below
%include /tmp/partition_layout

I am happy to report that for the servers that I have tested thus far, this method works. Is it possible that there will be primary and secondary disks that are identified via different path IDs not caught by my pre-installation script? Yes, that is possible. It’s also possible that your situation will require a completely different pre-installation script to gather the unique identifier of your choice. That being said, I hope that this post will help you devise your method of reliably identifying disks so that you can create the partition scheme automatically via your Kickstart configuration file. That way, you can then deploy it to many servers via PXE (or any other implementation that you’re currently utilising for mass deployment).


Robin Johnson a.k.a. robbat2 (homepage, bugs)
gnupg-2.1 mutt (June 05, 2015, 17:25 UTC)

For the mutt users with GnuPG, depending on your configuration, you might notice that mutt's handling of GnuPG mail stopped working with GnuPG. There were a few specific cases that would have caused this, which I'll detail, but if you just want it to work again, put the below into your Muttrc, and make the tweak to gpg-agent.conf. The underlying cause for most if it is that secret key operations have moved to the agent, and many Mutt users used the agent-less mode, because Mutt handled the passphrase nicely on it's own.

  • -u must now come BEFORE --cleansign
  • Add allow-loopback-pinentry to gpg-agent.conf, and restart the agent
  • The below config adds --pinentry-mode loopback before --passphrase-fd 0, so that GnuPG (and the agent) will accept it from Mutt still.
  • --verbose is optional, depending what you're doing, you might find --no-verbose cleaner.
  • --trust-model always is a personal preference for my Mutt mail usage, because I do try and curate my keyring
set pgp_autosign = yes
set pgp_use_gpg_agent = no
set pgp_timeout = 600
set pgp_sign_as="(your key here)"
set pgp_ignore_subkeys = no

set pgp_decode_command="gpg %?p?--pinentry-mode loopback  --passphrase-fd 0? --verbose --no-auto-check-trustdb --batch --output - %f"
set pgp_verify_command="gpg --pinentry-mode loopback --verbose --batch --output - --no-auto-check-trustdb --verify %s %f"
set pgp_decrypt_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - %f"
set pgp_sign_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - --armor --textmode %?a?-u %a? --detach-sign %f"
set pgp_clearsign_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - --armor --textmode %?a?-u %a? --detach-sign %f"
set pgp_encrypt_sign_command="pgpewrap gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --textmode --trust-model always --output - %?a?-u %a? --armor --encrypt --sign --armor -- -r %r -- %f"
set pgp_encrypt_only_command="pgpewrap gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --trust-model always --output --output - --encrypt --textmode --armor -- -r %r -- %f"
set pgp_import_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --import -v %f"
set pgp_export_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --export --armor %r"
set pgp_verify_key_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --fingerprint --check-sigs %r"
set pgp_list_pubring_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --with-colons --list-keys %r"
set pgp_list_secring_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --with-colons --list-secret-keys %r"

This entry was originally posted at Please comment there using OpenID.

May 26, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)

In my previous post regarding the migration of our production cluster to mongoDB 3.0 WiredTiger, we successfully upgraded all the secondaries of our replica-sets with decent performances and (almost, read on) no breakage.

Step 2 plan

The next step of our migration was to test our work load on WiredTiger primaries. After all, this is where the new engine would finally demonstrate all its capabilities.

  • We thus scheduled a step down from our 3.0 MMAPv1 primary servers so that our WiredTiger secondaries would take over.
  • Not migrating the primaries was a safety net in case something went wrong… And boy it went so wrong we’re glad we played it safe that way !
  • We rolled back after 10 minutes of utter bitterness.

The failure

After all the wait and expectation, I can’t express our level of disappointment at work when we saw that the WiredTiger engine could not handle our work load. Our application started immediately to throw 50 to 250 WriteConflict errors per minute !

Turns out that we are affected by this bug and that, of course, we’re not the only ones. So far it seems that it affects collections with :

  • heavy insert / update work loads
  • an unique index (or compound index)

The breakages

We also discovered that we’re affected by a weird mongodump new behaviour where the dumped BSON file does not contain the number of documents that mongodump said it was exporting. This is clearly a new problem because it happened right after all our secondaries switched to WiredTiger.

Since we have to ensure a strong consistency of our exports and that the mongoDB guys don’t seem so keen on moving on the bug (which I surely can understand) there is a large possibility that we’ll have to roll back even the WiredTiger secondaries altogether.

Not to mention that since the 3.0 version, we experience some CPU overloads crashing the entire server on our MMAPv1 primaries that we’re still trying to tackle before opening another JIRA bug…

Sad panda

Of course, any new major release such as 3.0 causes its headaches and brings its lot of bugs. We were ready for this hence the safety steps we took to ensure that we could roll back on any problem.

But as a long time advocate of mongoDB I must admit my frustration, even more after the time it took to get this 3.0 out and all the expectations that came with it.

I hope I can share some better news on the next blog post.

May 25, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)

I have been running a PostgreSQL cluster for a while as the primary backend for many services. The database system is very robust, well supported by the community and very powerful. In this post, I’m going to show how I use central authentication and authorization with PostgreSQL.

Centralized management is an important principle whenever deployments become very dispersed. For authentication and authorization, having a high-available LDAP is one of the more powerful components in any architecture. It isn’t the only method though – it is also possible to use a distributed approach where the master data is centrally managed, but the proper data is distributed to the various systems that need it. Such a distributed approach allows for high availability without the need for a highly available central infrastructure (user ids, group membership and passwords are distributed to the servers rather than queried centrally). Here, I’m going to focus on a mixture of both methods: central authentication for password verification, and distributed authorization.

PostgreSQL default uses in-database credentials

By default, PostgreSQL uses in-database credentials for the authentication and authorization. When a CREATE ROLE (or CREATE USER) command is issued with a password, it is stored in the pg_catalog.pg_authid table:

postgres# select rolname, rolpassword from pg_catalog.pg_authid;
    rolname     |             rolpassword             
 postgres_admin | 
 dmvsl          | 
 johan          | 
 hdc_owner      | 
 hdc_reader     | 
 hdc_readwrite  | 
 hadoop         | 
 swift          | 
 sean           | 
 hdpreport      | 
 postgres       | md5c127bc9fc185daf0e06e785876e38484

Authorizations are also stored in the database (and unless I’m mistaken, this cannot be moved outside):

postgres# \l db_hadoop
                                   List of databases
   Name    |   Owner   | Encoding |  Collate   |   Ctype    |     Access privileges     
 db_hadoop | hdc_owner | UTF8     | en_US.utf8 | en_US.utf8 | hdc_owner=CTc/hdc_owner  +
           |           |          |            |            | hdc_reader=c/hdc_owner   +
           |           |          |            |            | hdc_readwrite=c/hdc_owner

Furthermore, PostgreSQL has some additional access controls through its pg_hba.conf file, in which the access towards the PostgreSQL service itself can be governed based on context information (such as originating IP address, target database, etc.).

For more information about the standard setups for PostgreSQL, definitely go through the official PostgreSQL documentation as it is well documented and kept up-to-date.

Now, for central management, in-database settings become more difficult to handle.

Using PAM for authentication

The first step to move the management of authentication and authorization outside the database is to look at a way to authenticate users (password verification) outside the database. I tend not to use a distributed password approach (where a central component is responsible for changing passwords on multiple targets), instead relying on a high-available LDAP setup, but with local caching (to catch short-lived network hick-ups) and local password use for last-hope accounts (such as root and admin accounts).

PostgreSQL can be configured to directly interact with an LDAP, but I like to use Linux PAM whenever I can. For my systems, it is a standard way of managing the authentication of many services, so the same goes for PostgreSQL. And with the sys-auth/pam_ldap package integrating multiple services with LDAP is a breeze. So the first step is to have PostgreSQL use PAM for authentication. This is handled through its pg_hba.conf file:

# TYPE  DATABASE        USER    ADDRESS         METHOD          [OPTIONS]
local   all             all                     md5
host    all             all     all             pam             pamservice=postgresql

This will have PostgreSQL use the postgresql PAM service for authentication. The PAM configuration is thus in /etc/pam.d/postgresql. In it, we can either directly use the LDAP PAM modules, or use the SSSD modules and have SSSD work with LDAP.

Yet, this isn’t sufficient. We still need to tell PostgreSQL which users can be authenticated – the users need to be defined in the database (just without password credentials because that is handled externally now). This is done together with the authorization handling.

Users and group membership

Every service on the systems I maintain has dedicated groups in which for instance its administrators are listed. For instance, for the PostgreSQL services:

# getent group gpgsqladmin

A local batch job (ran through cron) queries this group (which I call the masterlist, as well as queries which users in PostgreSQL are assigned the postgres_admin role (which is a superuser role like postgres and is used as the intermediate role to assign to administrators of a PostgreSQL service), known as the slavelist. Delta’s are then used to add the user or remove it.

# Note: membersToAdd / membersToRemove / _psql are custom functions
#       so do not vainly search for them on your system ;-)
for member in $(membersToAdd ${masterlist} ${slavelist}) ; do
  _psql "CREATE USER ${member} LOGIN INHERIT;" postgres
  _psql "GRANT postgres_admin TO ${member};" postgres

for member in $(membersToRemove ${masterlist} ${slavelist}) ; do
  _psql "REVOKE postgres_admin FROM ${member};" postgres
  _psql "DROP USER ${member};" postgres

The postgres_admin role is created whenever I create a PostgreSQL instance. Likewise, for databases, a number of roles are added as well. For instance, for the db_hadoop database, the hdc_owner, hdc_reader and hdc_readwrite roles are created with the right set of privileges. Users are then granted this role if they belong to the right group in the LDAP. For instance:

# getent group gpgsqlhdc_own

With this simple approach, granting users access to a database is a matter of adding the user to the right group (like gpgsqlhdc_ro for read-only access to the Hadoop related database(s)) and either wait for the cron-job to add it, or manually run the authorization synchronization. By standardizing on infrastructural roles (admin, auditor) and data roles (owner, rw, ro) managing multiple databases is a breeze.

May 23, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs)


Die Troisdorfer Linux User Group (kurz TroLUG) veranstaltet

am Samstag den 01.08.2015
in Troisdorf nahe Köln/Bonn

einen Gentoo-Workshop, der sich an fortgeschrittene User richtet.

Mehr Details, die genaue Adresse und der Ablauf finden sich auf der entsprechenden Seite der TroLUG.




Alexys Jacob a.k.a. ultrabug (homepage, bugs)

Good news for gevent users blocked on python < 2.7.9 due to broken SSL support since python upstream dropped the private API _ssl.sslwrap that eventlet was using.

This issue was starting to get old and problematic since GLSA 2015-0310 but I’m happy to say that almost 6 hours after the gevent-1.0.2 release, it is already available on portage !

We were also affected by this issue at work so I’m glad that the tension between ops and devs this issue was causing will finally be over ;)

May 22, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)
Three new quotes to the database (May 22, 2015, 18:19 UTC)

Today I added three new quotes to the “My Favourite Quotes” database that you see on the right-hand side of my blog.

“The evil people of the world would have little impact without the sheep.”
–Tim Clark

“Apologising does not always mean that you’re wrong and the other person is right. It just means that you value your relationship more than your ego.”

“Some day soon, perhaps in forty years, there will be no one alive who has ever known me. That’s when I will be truly dead – when I exist in no one’s memory. I thought a lot about how someone very old is the last living individual to have known some person or cluster of people. When that person dies, the whole cluster dies,too, vanishes from the living memory. I wonder who that person will be for me. Whose death will make me truly dead?”
–Irvin D. Yalom in Love’s Executioner and Other Tales of Psychotherapy

The first one came up during a conversation that I was having with a colleague who is absolutely brilliant. That quote in particular sums up a very simplistic idea, but one that would have a dramatic impact on human existence if we were to recognise our sometimes blind following of others.


May 20, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)


My tech articles—especially Linux ones—are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Late last night, I decided to apply some needed updates to my personal mail server, which is running Gentoo Linux (OpenRC) with a mail stack of Postfix & Dovecot with AMaViS (filtering based on SpamAssassin, ClamAV, and Vipul’s Razor). After applying the updates, and restarting the necessary components of the mail stack, I ran my usual test of sending an email from one of my accounts to another one. It went through without a problem.

However, I realised that it isn’t a completely valid test to send an email from one internal account to another because I have amavisd configured to not scan anything coming from my trusted IPs and domains. I noticed several hundred mails in the queue when I ran postqueue -p, and they all had notices similar to:

status=deferred (delivery temporarily suspended:
connect to[]:10024: Connection refused)

That indicated to me that it wasn’t a problem with Postfix (and I knew it wasn’t a problem with Dovecot, because I could connect to my accounts via IMAP). Seeing as amavisd is running on localhost:10024, I figured that that is where the problem had to be. A lot of times, when there is a “connection refused” notification, it is because no service is listening on that port. You can test to see what ports are in a listening state and what processes, applications, or daemons are listening by running:

netstat -tupan | grep LISTEN

When I did that, I noticed that amavisd wasn’t listening on port 10024, which made me think that it wasn’t running at all. That’s when I ran into the strange part of the problem: the init script output:

# /etc/init.d/amavisd start
* WARNING: amavisd has already been started
# /etc/init.d/amavisd stop
The amavisd daemon is not running                [ !! ]
* ERROR: amavisd failed to start

So, apparently it is running and not running at the same time (sounds like a Linux version of Schrödinger’s cat to me)! It was obvious, though, that it wasn’t actually running (which could be verified with ‘ps -elf | grep -i amavis’). So, what to do? I tried manually removing the PID file, but that actually just made matters a bit worse. Ultimately, this combination is what fixed the problem for me:

/etc/init.d/amavisd zap
/etc/init.d/amavisd start

It seems that the SpamAssassin rules file had gone missing, and that was causing amavisd to not start properly. Manually updating the rules file (with ‘sa-update’) regenerated it, and then I zapped amavisd completely, and lastly restarted the daemon.

Hope that helps anyone running into the same problem.


EDIT: I have included a new post about debugging amavisd start-up problems.

Arun Raghavan a.k.a. ford_prefect (homepage, bugs)
GNOME Asia 2015 (May 20, 2015, 08:08 UTC)

I was in Depok, Indonesia last week to speak at GNOME Asia 2015. It was a great experience — the organisers did a fantastic job and as a bonus, the venue was incredibly pretty!

View from our room

View from our room

My talk was about the GNOME audio stack, and my original intention was to talk a bit about the APIs, how to use them, and how to choose which to use. After the first day, though, I felt like a more high-level view of the pieces would be more useful to the audience, so I adjusted the focus a bit. My slides are up here.

Nirbheek and I then spent a couple of days going down to Yogyakarta to cycle around, visit some temples, and sip some fine hipster coffee.

All in all, it was a week well spent. I’d like to thank the GNOME Foundation for helping me get to the conference!

Sponsored by GNOME!

Sponsored by GNOME!

May 19, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Well, it was that time of the year again this past weekend–time for the 2015 St. Louis Children’s Hospital Make Tracks for the Zoo race! This race is always one of my favourite ones of the year, so I was excited to partake in it. There’s always a great turnout (~3500 this year, I believe, but 1644 chipped for the 5K run), and lots of fun, family-oriented activities. One of the best parts is that there are people of all ages, and walks of life (pun intended) who show up to either walk or run to support the amazing Saint Louis Zoo!

2015 St. Louis Make Tracks for the Zoo pre-race line-up
2015 St. Louis Make Tracks for the Zoo – Pre-race
(click to enlarge)

It started out a little iffy this year because there was some substantial rain. Ultimately, though, the rain cleared and the sun started shining in time for the race. The only downside was that the course was pretty slick due to the rain. Despite the slippery course, I did better than expected in terms of time. My total 5K time was 19’06”, according to the chip-timing provided by Big River Running Company (which is a great local running store here in the Saint Louis area). You can see the full results, and sort by whatever criteria you would like by visiting their ChronoTrack site for this race.

2015 St. Louis Make Tracks for the Zoo results
2015 St. Louis Make Tracks for the Zoo results
(click to enlarge)

There were some great runners this year! My time earned me 13th overall, 11th in overall male, and 3rd in my bracket (which is now Male 30-34 [ugh, can't believe I turned 30 this month]). Some really outstanding participants (besides the obvious of the overall winners) this year were Zachary Niemeyer, who at 14 finished 15th overall with a great time of 19’10”, and Norman Jamieson, who at age 79 finished in just over 33 minutes! I’ll be lucky to be able to walk at that age! :-)

2015 St. Louis Make Tracks for the Zoo - Nathan Zachary
Me, tired and sweaty from the humidity, but happy with the experience
(click to enlarge)

So, it was another great year for Make Tracks (especially since it was the 30th year for it), and I’m already looking forward to next year’s race. Hopefully you’ll come and join in a fun time that supports a great establishment!


P.S. A special thanks to my Pops for not only coming out to support me, but also for snapping the photos above!

May 18, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Testing with permissive domains (May 18, 2015, 11:40 UTC)

When testing out new technologies or new setups, not having (proper) SELinux policies can be a nuisance. Not only are the number of SELinux policies that are available through the standard repositories limited, some of these policies are not even written with the same level of confinement that an administrator might expect. Or perhaps the technology to be tested is used in a completely different manner.

Without proper policies, any attempt to start such a daemon or application might or will cause permission violations. In many cases, developers or users tend to disable SELinux enforcing then so that they can continue playing with the new technology. And why not? After all, policy development is to be done after the technology is understood.

But completely putting the system in permissive mode is overshooting. It is much easier to make a very simple policy to start with, and then mark the domain as a permissive domain. What happens is that the software then, after transitioning into the “simple” domain, is not part of the SELinux enforcements anymore whereas the rest of the system remains in SELinux enforcing mode.

For instance, create a minuscule policy like so:

policy_module(testdom, 1.0)

type testdom_t;
type testdom_exec_t;
init_daemon_domain(testdom_t, testdom_exec_t)

Mark the executable for the daemon as testdom_exec_t (after building and loading the minuscule policy):

~# chcon -t testdom_exec_t /opt/something/bin/daemond

Finally, tell SELinux that testdom_t is to be seen as a permissive domain:

~# semanage permissive -a testdom_t

When finished, don’t forget to remove the permissive bit (semanage permissive -d testdom_t) and unload/remove the SELinux policy module.

And that’s it. If the daemon is now started (through a standard init script) it will run as testdom_t and everything it does will be logged, but not enforced by SELinux. That might even help in understanding the application better.

May 17, 2015
Hanno Böck a.k.a. hanno (homepage, bugs)

Keystl;dr News about a broken 4096 bit RSA key are not true. It is just a faulty copy of a valid key.

Earlier today a blog post claiming the factoring of a 4096 bit RSA key was published and quickly made it to the top of Hacker News. The key in question was the PGP key of a well-known Linux kernel developer. I already commented on Hacker News why this is most likely wrong, but I thought I'd write up some more details. To understand what is going on I have to explain some background both on RSA and on PGP keyservers. This by itself is pretty interesting.

RSA public keys consist of two values called N and e. The N value, called the modulus, is the interesting one here. It is the product of two very large prime numbers. The security of RSA relies on the fact that these two numbers are secret. If an attacker would be able to gain knowledge of these numbers he could use them to calculate the private key. That's the reason why RSA depends on the hardness of the factoring problem. If someone can factor N he can break RSA. For all we know today factoring is hard enough to make RSA secure (at least as long as there are no large quantum computers).

Now imagine you have two RSA keys, but they have been generated with bad random numbers. They are different, but one of their primes is the same. That means we have N1=p*q1 and N2=p*q2. In this case RSA is no longer secure, because calculating the greatest common divisor (GCD) of two large numbers can be done very fast with the euclidean algorithm, therefore one can calculate the shared prime value.

It is not only possible to break RSA keys if you have two keys with one shared factors, it is also possible to take a large set of keys and find shared factors between them. In 2012 Arjen Lenstra and his team published a paper using this attack on large scale key sets and at the same time Nadia Heninger and a team at the University of Michigan independently also performed this attack. This uncovered a lot of vulnerable keys on embedded devices, but these were mostly SSH and TLS keys. Lenstra's team however also found two vulnerable PGP keys. For more background you can watch this 29C3 talk by Nadia Heninger, Dan Bernstein and Tanja Lange.

PGP keyservers have been around since quite some time and they have a property that makes them especially interesting for this kind of research: They usually never delete anything. You can add a key to a keyserver, but you cannot remove it, you can only mark it as invalid by revoking it. Therefore using the data from the keyservers gives you a large set of cryptographic keys.

Okay, so back to the news about the supposedly broken 4096 bit key: There is a service called Phuctor where you can upload a key and it'll check it against a set of known vulnerable moduli. This service identified the supposedly vulnerable key.

The key in question has the key id e99ef4b451221121 and belongs to the master key bda06085493bace4. Here is the vulnerable modulus:

c844a98e3372d67f 562bd881da8ea66c a71df16deab1541c e7d68f2243a37665 c3f07d3dd6e651cc d17a822db5794c54 ef31305699a6c77c 043ac87cafc022a3 0a2a717a4aa6b026 b0c1c818cfc16adb aae33c47b0803152 f7e424b784df2861 6d828561a41bdd66 bd220cb46cd288ce 65ccaf9682b20c62 5a84ef28c63e38e9 630daa872270fa15 80cb170bfc492b80 6c017661dab0e0c9 0a12f68a98a98271 82913ff626efddfb f8ae8f1d40da8d13 a90138686884bad1 9db776bb4812f7e3 b288b47114e486fa 2de43011e1d5d7ca 8daf474cb210ce96 2aafee552f192ca0 32ba2b51cfe18322 6eb21ced3b4b3c09 362b61f152d7c7e6 51e12651e915fc9f 67f39338d6d21f55 fb4e79f0b2be4d49 00d442d567bacf7b 6defcd5818b050a4 0db6eab9ad76a7f3 49196dcc5d15cc33 69e1181e03d3b24d a9cf120aa7403f40 0e7e4eca532eac24 49ea7fecc41979d0 35a8e4accea38e1b 9a33d733bea2f430 362bd36f68440ccc 4dc3a7f07b7a7c8f cdd02231f69ce357 4568f303d6eb2916 874d09f2d69e15e6 33c80b8ff4e9baa5 6ed3ace0f65afb43 60c372a6fd0d5629 fdb6e3d832ad3d33 d610b243ea22fe66 f21941071a83b252 201705ebc8e8f2a5 cc01112ac8e43428 50a637bb03e511b2 06599b9d4e8e1ebc eb1e820d569e31c5 0d9fccb16c41315f 652615a02603c69f e9ba03e78c64fecc 034aa783adea213b

In fact this modulus is easily factorable, because it can be divided by 3. However if you look at the master key bda06085493bace4 you'll find another subkey with this modulus:

c844a98e3372d67f 562bd881da8ea66c a71df16deab1541c e7d68f2243a37665 c3f07d3dd6e651cc d17a822db5794c54 ef31305699a6c77c 043ac87cafc022a3 0a2a717a4aa6b026 b0c1c818cfc16adb aae33c47b0803152 f7e424b784df2861 6d828561a41bdd66 bd220cb46cd288ce 65ccaf9682b20c62 5a84ef28c63e38e9 630daa872270fa15 80cb170bfc492b80 6c017661dab0e0c9 0a12f68a98a98271 82c37b8cca2eb4ac 1e889d1027bc1ed6 664f3877cd7052c6 db5567a3365cf7e2 c688b47114e486fa 2de43011e1d5d7ca 8daf474cb210ce96 2aafee552f192ca0 32ba2b51cfe18322 6eb21ced3b4b3c09 362b61f152d7c7e6 51e12651e915fc9f 67f39338d6d21f55 fb4e79f0b2be4d49 00d442d567bacf7b 6defcd5818b050a4 0db6eab9ad76a7f3 49196dcc5d15cc33 69e1181e03d3b24d a9cf120aa7403f40 0e7e4eca532eac24 49ea7fecc41979d0 35a8e4accea38e1b 9a33d733bea2f430 362bd36f68440ccc 4dc3a7f07b7a7c8f cdd02231f69ce357 4568f303d6eb2916 874d09f2d69e15e6 33c80b8ff4e9baa5 6ed3ace0f65afb43 60c372a6fd0d5629 fdb6e3d832ad3d33 d610b243ea22fe66 f21941071a83b252 201705ebc8e8f2a5 cc01112ac8e43428 50a637bb03e511b2 06599b9d4e8e1ebc eb1e820d569e31c5 0d9fccb16c41315f 652615a02603c69f e9ba03e78c64fecc 034aa783adea213b

You may notice that these look pretty similar. But they are not the same. The second one is the real subkey, the first one is just a copy of it with errors.

If you run a batch GCD analysis on the full PGP key server data you will find a number of such keys (Nadia Heninger has published code to do a batch GCD attack). I don't know how they appear on the key servers, I assume they are produced by network errors, harddisk failures or software bugs. It may also be that someone just created them in some experiment.

The important thing is: Everyone can generate a subkey to any PGP key and upload it to a key server. That's just the way the key servers work. They don't check keys in any way. However these keys should pose no threat to anyone. The only case where this could matter would be a broken implementation of the OpenPGP key protocol that does not check if subkeys really belong to a master key.

However you won't be able to easily import such a key into your local GnuPG installation. If you try to fetch this faulty sub key from a key server GnuPG will just refuse to import it. The reason is that every sub key has a signature that proves that it belongs to a certain master key. For those faulty keys this signature is obviously wrong.

Now here's my personal tie in to this story: Last year I started a project to analyze the data on the PGP key servers. And at some point I thought I had found a large number of vulnerable PGP keys – including the key in question here. In a rush I wrote a mail to all people affected. Only later I found out that something was not right and I wrote to all affected people again apologizing. Most of the keys I thought I had found were just faulty keys on the key servers.

The code I used to parse the PGP key server data is public, I also wrote a background paper and did a talk at the BsidesHN conference.

May 14, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Well, sometimes there is a mistake so blatantly obvious on a website that it just has to be shared with others. I was looking at the Material Safety Data Sheets from Exxon Mobil (don’t ask; I was just interested in seeing the white paper on Mobil jet oil), and I noticed something rather strange in the drop-down menu:

Click for a larger version

Yup, apparently now Africa is a country. Funny, all this time I thought it was a continent. Interestingly enough, you can see that they did list Angola and Botswana as countries. Countries within countries—either very meta, or along the lines of country-ception (oh the double entendre), but I digress…


May 13, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
uWSGI, gevent and pymongo 3 threads mayhem (May 13, 2015, 14:56 UTC)

This is a quick heads-up post about a behaviour change when running a gevent based application using the new pymongo 3 driver under uWSGI and its gevent loop.

I was naturally curious about testing this brand new and major update of the python driver for mongoDB so I just played it dumb : update and give a try on our existing code base.

The first thing I noticed instantly is that a vast majority of our applications were suddenly unable to reload gracefully and were force killed by uWSGI after some time !

worker 1 (pid: 9839) is taking too much time to die...NO MERCY !!!

uWSGI’s gevent-wait-for-hub

All our applications must be able to be gracefully reloaded at any time. Some of them are spawning quite a few greenlets on their own so as an added measure of making sure we never loose any running greenlet we use the gevent-wait-for-hub option, which is described as follow :

wait for gevent hub's death instead of the control greenlet

… which does not mean a lot but is explained in a previous uWSGI changelog :

During shutdown only the greenlets spawned by uWSGI are taken in account,
and after all of them are destroyed the process will exit.

This is different from the old approach where the process wait for
ALL the currently available greenlets (and monkeypatched threads).

If you prefer the old behaviour just specify the option gevent-wait-for-hub

pymongo 3

Compared to its previous 2.x versions, one of the overall key aspect of the new pymongo 3 driver is its intensive usage of threads to handle server discovery and connection pools.

Now we can relate this very fact to the gevent-wait-for-hub behaviour explained above :

the process wait for ALL the currently available greenlets
(and monkeypatched threads)

This explained why our applications were hanging until the reload-mercy (force kill) timeout option of uWSGI hit the fan !


When using pymongo 3 with the gevent-wait-for-hub option, you have to keep in mind that all of pymongo’s threads (so monkey patched threads) are considered as active greenlets and will thus be waited for termination before uWSGI recycles the worker !

Two options come in mind to handle this properly :

  1. stop using the gevent-wait-for-hub option and change your code to use a gevent pool group to make sure that all of your important greenlets are taken care of when a graceful reload happens (this is how we do it today, the gevent-wait-for-hub option usage was just over protective for us).
  2. modify your code to properly close all your pymongo connections on graceful reloads.

Hope this will save some people the trouble of debugging this ;)

May 10, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Audit buffering and rate limiting (May 10, 2015, 12:18 UTC)

Be it because of SELinux experiments, or through general audit experiments, sometimes you’ll get in touch with a message similar to the following:

audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=44395 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded

The message shows up when certain audit events could not be logged through the audit subsystem. Depending on the system configuration, they might be either ignored, sent through the kernel logging infrastructure or even have the system panic. And if the messages are sent to the kernel log then they might show up, but even that log has its limitations, which can lead to output similar to the following:

__ratelimit: 53 callbacks suppressed

In this post, I want to give some pointers in configuring the audit subsystem as well as understand these messages…

There is auditd and kauditd

If you take a look at the audit processes running on the system, you’ll notice that (assuming Linux auditing is used of course) two processes are running:

# ps -ef | grep audit
root      1483     1  0 10:11 ?        00:00:00 /sbin/auditd
root      1486     2  0 10:11 ?        00:00:00 [kauditd]

The /sbin/auditd daemon is the user-space audit daemon. It registers itself with the Linux kernel audit subsystem (through the audit netlink system), which responds with spawning the kauditd kernel thread/process. The fact that the process is a kernel-level one is why the kauditd is surrounded by brackets in the ps output.

Once this is done, audit messages are communicated through the netlink socket to the user-space audit daemon. For the detail-oriented people amongst you, look for the kauditd_send_skb() method in the kernel/audit.c file. Now, generated audit event messages are not directly relayed to the audit daemon – they are first queued in a sort-of backlog, which is where the backlog-related messages above come from.

Audit backlog queue

In the kernel-level audit subsystem, a socket buffer queue is used to hold audit events. Whenever a new audit event is received, it is logged and prepared to be added to this queue. Adding to this queue can be controlled through a few parameters.

The first parameter is the backlog limit. Be it through a kernel boot parameter (audit_backlog_limit=N) or through a message relayed by the user-space audit daemon (auditctl -b N), this limit will ensure that a queue cannot grow beyond a certain size (expressed in the amount of messages). If an audit event is logged which would grow the queue beyond this limit, then a failure occurs and is handled according to the system configuration (more on that later).

The second parameter is the rate limit. When more audit events are logged within a second than set through this parameter (which can be controlled through a message relayed by the user-space audit system, using auditctl -r N) then those audit events are not added to the queue. Instead, a failure occurs and is handled according to the system configuration.

Only when the limits are not reached is the message added to the queue, allowing the user-space audit daemon to consume those events and log those according to the audit configuration. There are some good resources on audit configuration available on the Internet. I find this SuSe chapter worth reading, but many others exist as well.

There is a useful command related to the subject of the audit backlog queue. It queries the audit subsystem for its current status:

# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=1483 rate_limit=0 backlog_limit=8192 lost=3 backlog=0

The command displays not only the audit state (enabled or not) but also the settings for rate limits (on the audit backlog) and backlog limit. It also shows how many events are currently still waiting in the backlog queue (which is zero in our case, so the audit user-space daemon has properly consumed and logged the audit events).

Failure handling

If an audit event cannot be logged, then this failure needs to be resolved. The Linux audit subsystem can be configured do either silently discard the message, switch to the kernel log subsystem, or panic. This can be configured through the audit user-space (auditctl -f [0..2]), but is usually left at the default (which is 1, being to switch to the kernel log subsystem).

Before that is done, the message is displayed which reveals the cause of the failure handling:

audit: audit_backlog=321 > audit_backlog_limit=320
audit: audit_lost=44395 audit_rate_limit=0 audit_backlog_limit=320
audit: backlog limit exceeded

In this case, the backlog queue was set to contain at most 320 entries (which is low for a production system) and more messages were being added (the Linux kernel in certain cases allows to have a few more entries than configured for performance and consistency reasons). The number of events already lost is displayed, as well as the current limitation settings. The message “backlog limit exceeded” can be “rate limit exceeded” if that was the limitation that was triggered.

Now, if the system is not configured to silently discard it, or to panic the system, then the “dropped” messages are sent to the kernel log subsystem. The calls however are also governed through a configurable limitation: it uses a rate limit which can be set through sysctl:

# sysctl -a | grep kernel.printk_rate
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10

In the above example, this system allows one message every 5 seconds, but does allow a burst of up to 10 messages at once. When the rate limitation kicks in, then the kernel will log (at most one per second) the number of suppressed events:

[40676.545099] __ratelimit: 246 callbacks suppressed

Although this limit is kernel-wide, not all kernel log events are governed through it. It is the caller subsystem (in our case, the audit subsystem) which is responsible for having its events governed through this rate limitation or not.

Finishing up

Before waving goodbye, I would like to point out that the backlog queue is a memory queue (and not on disk, Red Hat), just in case it wasn’t obvious. Increasing the queue size can result in more kernel memory consumption. Apparently, a practical size estimate is around 9000 bytes per message. On production systems, it is advised not to make this setting too low. I personally set it to 8192.

Lost audit events might result in difficulties for troubleshooting, which is the case when dealing with new or experimental SELinux policies. It would also result in missing security-important events. It is the audit subsystem, after all. So tune it properly, and enjoy the power of Linux’ audit subsystem.

May 09, 2015
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)
Reviving Gentoo VMware support (May 09, 2015, 23:41 UTC)

Sadly over the last months the support for VMware Workstation and friends in Gentoo dropped a lot. Why? Well, I was the only developer left who cared, and it's definitely not at the top of my Gentoo priorities list. To be honest that has not really changed. However... let's try to harness the power of the community now.

I've pushed a mirror of the Gentoo vmware overlay to Github, see

If you have improvements, version bumps, ... - feel free to generate pull requests. Everything related to VMware products is acceptable. I hope some more people will over time sign up and help merging. Just be careful when using the overlay, it likely won't get the same level of review as ebuilds in the main tree.

May 07, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)

We’ve been running a nice mongoDB cluster in production for several years now in my company.

This cluster suits quite a wide range of use cases from very simple configuration collections to complex queried ones and real time analytics. This versatility has been the strong point of mongoDB for us since the start as it allows different teams to address their different problems using the same technology. We also run some dedicated replica sets for other purposes and network segmentation reasons.

We’ve waited a long time to see the latest 3.0 release features happening. The new WiredTiger storage engine hit the fan at the right time for us since we had reached the limits of our main production cluster and were considering alternatives.

So as surprising it may seem, it’s the first of our mongoDB architecture we’re upgrading to v3.0 as it has become a real necessity.

This post is about sharing our first experience about an ongoing and carefully planned major upgrade of a production cluster and does not claim to be a definitive migration guide.

Upgrade plan and hardware

The upgrade process is well covered in the mongoDB documentation already but I will list the pre-migration base specs of every node of our cluster.

  • mongodb v2.6.8
  • RAID1 spinning HDD 15k rpm for the OS (Gentoo Linux)
  • RAID10 4x SSD for mongoDB files under LVM
  • 64 GB RAM

Our overall philosophy is to keep most of the configuration parameters to their default values to start with. We will start experimenting with them when we have sufficient metrics to compare with later.

Disk (re)partitioning considerations

The master-get-all-the-writes architecture is still one of the main limitation of mongoDB and this does not change with v3.0 so you obviously need to challenge your current disk layout to take advantage of the new WiredTiger engine.

mongoDB 2.6 MMAPv1

Considering our cluster data size, we were forced to use our 4 SSD in a RAID10 as it was the best compromise to preserve performance while providing sufficient data storage capacity.

We often reached the limits of our I/O and moved the journal out of the RAID10 to the mostly idle OS RAID1 with no significant improvements.

mongoDB 3.0 WiredTiger

The main consideration point for us is the new feature allowing to store the indexes in a separate directory. So we anticipated the data storage consumption reduction thanks to the snappy compression and decided to split our RAID10 in two dedicated RAID1.

Our test layout so far is :

  • RAID1 SSD for the data
  • RAID1 SSD for the indexes and journal

Our first node migration

After migrating our mongos and config servers to 3.0, we picked our worst performing secondary node to test the actual migration to WiredTiger. After all, we couldn’t do worse right ?

We are aware that the strong suit of WiredTiger is actually about having the writes directed to it and will surely share our experience of this aspect later.

compression is bliss

To make this comparison accurate, we resynchronized this node totally before migrating to WiredTiger so we could compare a non fragmented MMAPv1 disk usage with the WiredTiger compressed one.

While I can’t disclose the actual values, compression worked like a charm for us with a gain ratio of 3,2 on disk usage (data + indexes) which is way beyond our expectations !

This is the DB Storage graph from MMS, showing a gain ratio of 4 surely due to indexes being in a separate disk now.






memory usage

As with the disk usage, the node had been running hot on MMAPv1 before the actual migration so we can compare memory allocation/consumption of both engines.

There again the memory management of WiredTiger and its cache shows great improvement. For now, we left the default setting which has WiredTiger limit its cache to half the available memory of the system. We’ll experiment with this setting later on.







This I’m still not sure of the actual cause yet but the connections count is higher and more steady than before on this node.


First impressions

The node is running smooth for several hours now. We are getting acquainted to the new metrics and statistics from WiredTiger. The overall node and I/O load is better than before !

While all the above graphs show huge improvements there is no major change from our applications point of view. We didn’t expect any since this is only one node in a whole cluster and that the main benefits will also come from master node migrations.

I’ll continue to share our experience and progress about our mongoDB 3.0 upgrade.

May 06, 2015
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

It's time to announce yet another nice result. Our manuscript "Transport across a carbon nanotube quantum dot contacted with ferromagnetic leads: experiment and non-perturbative modeling" has been accepted as a regular article by Physical Review B.
When ferromagnetic materials are used as contacts for a carbon nanotube at low temperature, the current is strongly influenced by the direction of the contact magnetization via the so-called tunneling magnetoresistance (TMR). Since the nanotube contains a quantum dot, in addition its electronic energy levels play an important role; the TMR depends on the gate voltage value and can reach large negative and positive values. Here, in another fruitful joint experimental and theoretical effort, we present both measurements of the gate-dependent TMR across a "shell" of four Coulomb oscillations, and model them in the so-called "dressed second order" framework. The calculations nicely reproduce the characteristic oscillatory patterns of the TMR gate dependence.

"Transport across a carbon nanotube quantum dot contacted with ferromagnetic leads: experiment and non-perturbative modeling"
A. Dirnaichner, M. Grifoni, A. Prüfling, D. Steininger, A. K. Hüttel, and Ch. Strunk
Phys. Rev. B 91, 195402 (2015); arXiv:1502.02005 (PDF)

May 04, 2015
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

We're happy to be able to announce that our manuscript "Broken SU(4) symmetry in a Kondo-correlated carbon nanotube" has been accepted for publication in Physical Review B.
This manuscript is the result of a joint experimental and theoretical effort. We demonstrate that there is a fundamental difference between cotunneling and the Kondo effect - a distinction that has been debated repeatedly in the past. In carbon nanotubes, the two graphene-derived Dirac points can lead to a two-fold valley degeneracy in addition to spin degeneracy; each orbital "shell" of a confined electronic system can be filled with four electrons. In most nanotubes, these degeneracies are broken by the spin-orbit interaction (due to the wall curvature) and by valley mixing (due to, as recently demonstrated, scattering at the nanotube boundaries). Using an externally applied magnetic field, the quantum states involved in equilibrium (i.e., elastic, zero-bias) and nonequilibrium (i.e., inelastic, finite bias) transitions can be identified. We show theoretically and experimentally that in the case of Kondo correlations, not all quantum state pairs contribute to Kondo-enhanced transport; some of these are forbidden by symmetries stemming from the carbon nanotube single particle Hamiltonian. This is distinctly different from the case of inelastic cotunneling (at higher temperatures and/or weaker quantum dot-lead coupling), where all transitions have been observed in the past.

"Broken SU(4) symmetry in a Kondo-correlated carbon nanotube"
D. R. Schmid, S. Smirnov, M. Marganska, A. Dirnaichner, P. L. Stiller, M. Grifoni, A. K. Hüttel, and Ch. Strunk
Phys. Rev. B 91, 155435 (2015) (PDF)

Nirbheek Chauhan a.k.a. nirbheek (homepage, bugs)
A Transcoding Proxy for HTTP Video Streams (May 04, 2015, 07:43 UTC)

Sometime last year, we worked on a client project to create a prototype for a server that is, in essence, a "transcoding proxy". It accepts N HTTP client streams and makes them available for an arbitrary number of clients via HTTP GET (and/or over RTP/UDP) in the form of WebM streams. Basically, it's something similar to The terms of our work with the client allowed us to make this work available as Free and Open Source Software, and this blog post is the announcement of its public availability.

Go and try it out!

git clone

The purpose of this release is to demonstrate some of the streaming/live transcoding capabilities of GStreamer and the capabilities of LibSoup as an HTTP server library. Some details about the server follow, but there's more documentation and examples on how to use the server in the git repository.

In addition to using GStreamer, the server uses the GNOME HTTP library LibSoup for implementing the HTTP server which accepts and makes available live HTTP streams. Stress-testing for up to 100 simultaneous clients has been done with the server, with a measured end-to-end stream latency of between 150ms to 250ms depending on the number of clients. This can be likely improved by using codecs with lower latency and so on—after all the project is just a prototype. :)

The N client streams sent to the proxy via HTTP PUT/PUSH are transcoded to VP8/Vorbis WebM if needed, but are simply remuxed and passed through if they are in the same format. Optionally, the proxy can also broadcast each client stream to a list of pre-specified hosts via RTP/UDP.

Clients that want to stream video from the server can connect or disconnect at any time, and will get the current stream whenever they (re)connect. The server also accepts HTTP streams via both Chunked-Encoding and fixed-length HTTP PUT requests.

There is also a JSON-based REST API to interact with the server. There is also in-built validation via a Token Server. A specified host (or address mask) can be whitelisted, which will allow it to add or remove session id tokens along with details about the types of streams that the specified session id is allowed to send to the server, and the types of streams that will be made available by the proxy. For more information, see the REST API documentation.

We hope you find this example instructive in how to use LibSoup to implement an HTTP server and in using GStreamer for streaming and encoding purposes. Looking forward to hearing from you about it!

May 02, 2015
Hanno Böck a.k.a. hanno (homepage, bugs)

Google Password AlertA few days ago Google released a Chrome extension that emits a warning if a user types in his Google account password on a foreign webpage. This is meant as a protection against phishing pages. Code is on Github and the extension can be installed through Google's Chrome Web Store.

When I heard this the first time I already thought that there are probably multiple ways to bypass that protection with some Javascript trickery. Seems I was right. Shortly after the extension was released security researcher Paul Moore published a way to bypass the protection by preventing the popup from being opened. This was fixed in version 1.4.

At that point I started looking into it myself. Password Alert tries to record every keystroke from the user and checks if that matches the password (it doesn't store the password, only a hash). My first thought was to simulate keystrokes via Javascript. I have to say that my Javascript knowledge is close to nonexistent, but I can use Google and read Stackoverflow threads, so I came up with this:

function simkey(e) {
if (e.which==0) return;
var ev=document.createEvent("KeyboardEvent");
ev.initKeyboardEvent("keypress", true, true, window, 0, 0, 0, 0, 0, 0);
<form action="" method="POST">
<input type="password" id="pw" name="pw" onkeypress="simkey(event);">
<input type="submit">

For every key a user presses this generates a Javascript KeyboardEvent. This is enough to confuse the extension. I reported this to the Google Security Team and Andrew Hintz. Literally minutes before I sent the mail a change was committed that did some sanity checks on the events and thus prevented my bypass from working (it checks the charcode and it seems there is no way in webkit to generate a KeyboardEvent with a valid charcode).

While I did that Paul Moore also created another bypass which relies on page reloads. A new version 1.6 was released fixing both my and Moores bypass.

I gave it another try and after a couple of failures I came up with a method that still works. The extension will only store keystrokes entered on one page. So what I did is that on every keystroke I create a popup (with the already typed password still in the form field) and close the current window. The closing doesn't always work, I'm not sure why that's the case, this can probably be improved somehow. There's also some flickering in the tab bar. The password is passed via URL, this could also happen otherwise (converting that from GET to POST variable is left as an exercise to the reader). I'm also using PHP here to insert the variable into the form, this could be done in pure Javascript. Here's the code, still working with the latest version:

function rlt() {""+document.getElementById("pw").value);
<form action="." method="POST">
<input type="text" name="pw" id="pw" onkeyup="rlt();" onfocus="this.value=this.value;" value="<?php
if (isset($_GET['val'])) echo $_GET['val'];
<input type="submit">

Honestly I have a lot of doubts if this whole approach is a good idea. There are just too many ways how this can be bypassed. I know that passwords and phishing are a big problem, I just doubt this is the right approach to tackle it.

One more thing: When I first tested this extension I was confused, because it didn't seem to work. What I didn't know is that this purely relies on keystrokes. That means when you copy-and-paste your password (e. g. from some textfile in a crypto container) then the extension will provide you no protection. At least to me this was very unexpected behaviour.

Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
New personal gamestation! (May 02, 2015, 00:33 UTC)

Beside a new devbox that I talked about setting up, now that I no longer pay for the tinderbox I also decided to buy myself a new PC for playing games (so Windows-bound, unfortunately), replacing Yamato that has been serving me well for years at this point.

Given we've talked about this at the office as well, I'll write down the specs over here, with the links to Amazon (where I bought the components), as I know a fair number of people are always interested to know specs. I will probably write down some reviews on Amazon itself as well as on the blog, for the components that can be discussed "standalone".

  • CPU: Intel i7 5930K (Amazon UK), hex-core Haswell-E; it was intended as a good compromise between high performance and price, not only for gaming but also for Adobe Lightroom.
  • Motherboard: Asus X99-S (Amazon UK)
  • Memory: Crucial Ballistix 32GB (8GBx4) (Amazon UK) actually this one I ordered from Crucial directly, because the one I originally ordered on Amazon UK was going to ship from Las Vegas, which meant I had to pay customs on it. I am still waiting for that one to be fully cancelled, but then Crucial was able to deliver an order placed on Wednesday at 10pm by Friday, which was pretty good (given that this is a long weekend in Ireland.)
  • Case: Fractal Design Define R5 (Amazon UK) upon suggestion of two colleagues, one who only saw it in reviews, the other actually having the previous version. It is eerily quiet and very well organized; it would also fit a huge amount of storage if I needed to build a new NAS rather than a desktop PC.
  • CPU cooler: NZXT Kraken X61 (Amazon UK) I went with water cooling for the CPU because I did not like the size of the copper fins in the other alternatives of suggested coolers for the chosen CPU. Since this is a completely sealed system it didn't feel too bad. The only shaky part is that the only proper space for this to fit into the case is on the top-front side, and it does require forcing the last insulation panel in a little bit.

Now you probably notices some parts missing; the reason is that I have bought a bunch of components to upgrade Yamato over the past year and a half since being employed also means being able to just scratch your itch for power more easily, especially if you, like me, are single and not planning a future as a stock player. Some of the updates are still pretty good and others are a bit below average now, and barely average when I bought it, but I think it might be worth listing them still.

  • SSD: Samsung 850 EVO (Amazon UK) and Crucial M550 (Amazon UK), both 1TB. The reason for having two different ones is because the latter (which was the first of the two) was not available when I decided to get a second one, and the reason to get a second one was because I realized that while keeping pictures on the SSD helped a lot, the rest of the OS was still too slow…
  • GPU: Asus GeForce GTX660 (Amazon UK) because I needed something good that didn't cost too much at the time.
  • PSU: be quiet! Dark Power Pro 1200W (Amazon UK) which I had to replace when I bought the graphics card, as the one I had before didn't have the right PCI-E power connectors, or rather it had one too few. Given that Yamato is a Dual-Quad Opteron, with registered ECC memory, I needed something that would at least take 1kW; I'm not sure how much it's consuming right now to be honest.

We'll see how it fares once I have it fully installed and started playing games on it, I guess.

May 01, 2015
Hanno Böck a.k.a. hanno (homepage, bugs)
DNS AXFR scan data (May 01, 2015, 22:25 UTC)

I recently was made aware of an issue that many authoritative nameservers answer to AXFR requests. AXFR is a feature of the Domain Name System (DNS) that allows to query complete zones from a name server. That means one can find out all subdomains for a given domain.

If you want to see how this looks Verizon kindly provides you a DNS server that will answer with a very large zone to AXFR requests:
dig axfr

This by itself is not a security issue. It can however become a problem if people consider some of their subdomains / URLs secret. While checking this issue I found one example where such a subdomain contained a logging interface that exposed data that was certainly not meant to be public. However it is a bad idea in general to have "secret" subdomains, because there is no way to keep them really secret. DNS itself is not encrypted, therefore by sniffing your traffic it is always possible to see your "secret" subdomains.

AXFR is usually meant to be used between trusting name servers and requests by public IPs should not be answered. While it is in theory possible that someone considers publicly available AXFR a desired feature I assume in the vast majority these are just misconfigurations and were never intended to be public. I contacted a number of these and when they answered none of them claimed that this was an intended configuration. I'd generally say that it's wise to disable services you don't need. Recently US-CERT has issued an advisory about this issue.

I have made a scan of the Alexa top 1 million web pages and checked if their DNS server answers to AXFR requests. The University of Michigan has a project to collect data from Internet scans and I submitted my scan results to them. So you're welcome to download and analyze the data.

April 30, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)

If you are using SELinux on production systems (with which I mean systems that you offer services with towards customers or other parties beyond you, yourself and your ego), please consider proper change management if you don’t do already. SELinux is a very sensitive security subsystem – not in the sense that it easily fails, but because it is very fine-grained and as such can easily stop applications from running when their behavior changes just a tiny bit.

Sensitivity of SELinux

SELinux is a wonderful security measure for Linux systems that can prevent successful exploitation of vulnerabilities or misconfigurations. Of course, it is not the sole security measure that systems should take. Proper secure configuration of services, least privilege accounts, kernel-level mitigations such as grSecurity and more are other measures that certainly need to be taken if you really find system security to be a worthy goal to attain. But I’m not going to talk about those others right now. What I am going to focus on is SELinux, and how sensitive it is to changes.

An important functionality of SELinux to understand is that it segregates the security control system itself (the SELinux subsystem) from its configuration (the policy). The security control system itself is relatively small, and focuses on enforcement of the policy and logging (either because the policy asks to log something, or because something is prevented, or because an error occurred). The most difficult part of handling SELinux on a system is not enabling or interacting with it. No, it is its policy.

The policy is also what makes SELinux so darn sensitive for small system changes (or behavior that is not either normal, or at least not allowed through the existing policy). Let me explain with a small situation that I recently had.

Case in point: Switching an IP address

A case that beautifully shows how sensitive SELinux can be is an IP address change. My systems all obtain their IP address (at least for IPv4) from a DHCP system. This is of course acceptable behavior as otherwise my systems would never be able to boot up successfully anyway. The SELinux policy that I run also allows this without any hindrance. So that was not a problem.

Yet recently I had to switch an IP address for a system in production. All the services I run are set up in a dual-active mode, so I started with the change by draining the services to the second system, shutting down the service and then (after reconfiguring the DHCP system to now provide a different IP address) reload the network configuration. And then it happened – the DHCP client just stalled.

As the change failed, I updated the DHCP system again to deliver the old IP address and then reloaded the network configuration on the client. Again, it failed. Dumbstruck, I looked at the AVC denials and lo and behold, I notice a dig process running in a DHCP client related domain that is trying to do UDP binds, which the policy (at that time) did not allow. But why now suddenly, after all – this system was running happily for more than a year already (and with occasional reboots for kernel updates).

I won’t bore you with the investigation. It boils down to the fact that the DHCP client detected a change compared to previous startups, and was configured to run a few hooks as additional steps in the IP lease setup. As these hooks were never ran previously, the policy was never challenged to face this. And since the address change occurred a revert to the previous situation didn’t work either (as its previous state information was already deleted).

I was able to revert the client (which is a virtual guest in KVM) to the situation right before the change (thank you savevm and loadvm functionality) so that I could work on the policy first in a non-production environment so that the next change attempt was successful.

Change management

The previous situation might be “solved” by temporarily putting the DHCP client domain in permissive mode just for the change and then back. But that is ignoring the issue, and unless you have perfect operational documentation that you always read before making system or configuration changes, I doubt that you’ll remember this for the next time.

The case is also a good example on the sensitivity of SELinux. It is not just when software is being upgraded. Every change (be it in configuration, behavior or operational activity) might result in a situation that is new for the loaded SELinux policy. As the default action in SELinux is to deny everything, this will result in unexpected results on the system. Sometimes very visible (no IP address obtained), sometimes hidden behind some weird behavior (hostname correctly set but not the domainname) or perhaps not even noticed until far later. Compare it to the firewall rule configurations: you might be able to easily confirm that standard flows are still passed through, but how are you certain that fallback flows or one-in-a-month connection setups are not suddenly prevented from happening.

A somewhat better solution than just temporarily disabling SELinux access controls for a domain is to look into proper change management. Whenever a change has to be done, make sure that you

  • can easily revert the change back to the previous situation (backups!)
  • have tested the change on a non-vital (preproduction) system first

These two principles are pretty vital when you are serious about using SELinux in production. I’m not talking about a system that hardly has any fine-grained policies, like where most of the system’s services are running in “unconfined” domains (although that’s still better than not running with SELinux at all), but where you are truly trying to put a least privilege policy in place for all processes and services.

Being able to revert a change allows you to quickly get a service up and running again so that customers are not affected by the change (and potential issues) for long time. First fix the service, then fix the problem. If you are an engineer like me, you might rather focus on the problem (and a permanent, correct solution) first. But that’s wrong – always first make sure that the customers are not affected by it. Revert and put the service back up, and then investigate so that the next change attempt will not go wrong anymore.

Having a multi-master setup might give some more leeway into investigating issues (as the service itself is not disrupted) so in the case mentioned above I would probably have tried fixing the issue immediately anyway if it wasn’t policy-based. But most users do not have truly multi-master service setups.

Being able to test (and retest) changes in non-production also allows you to focus on automation (so that changes can be done faster and in a repeated, predictable and qualitative manner), regression testing as well as change accumulation testing.

You don’t have time for that?

Be honest with yourself. If you support services for others (be it in a paid-for manner or because you support an organization in your free time) you’ll quickly learn that service availability is one of the most qualitative aspects of what you do. No matter what mess is behind it, most users don’t see all that. All they see is the service itself (and its performance / features). If a change you wanted to make made a service unavailable for hours, users will notice. And if the change wasn’t communicated up front or it is the n-th time that this downtime occurs, they will start asking questions you rather not hear.

Using a non-production environment is not that much of an issue if the infrastructure you work with supports bare metal restores, or snapshot/cloning (in case of VMs). After doing those a couple of times, you’ll easily find that you can create a non-production environment from the production one. Or, you can go for a permanent non-production environment (although you’ll need to take care that this environment is at all times representative for the production systems).

And regarding qualitative changes, I really recommend to use a configuration management system. I recently switched from Puppet to Saltstack and have yet to use the latter to its fullest set (most of what I do is still scripted), but it is growing on me and I’m pretty convinced that I’ll have the majority of my change management scripts removed by the end of this year towards Saltstack-based configurations. And that’ll allow me to automate changes and thus provide a more qualitative service offering.

With SELinux, of course.

April 29, 2015
Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
Code Hygiene (April 29, 2015, 03:03 UTC)

Some convenient Makefile targets that make it very easy to keep code clean:

        scan-build clang foo.c -o foo

        indent -linux *.c
scan-build is llvm/clang's static analyzer and generates some decent warnings. Using clang to build (in addition to 'default' gcc in my case) helps diversity and sometimes catches different errors.

indent makes code pretty, the 'linux' default settings are not exactly what I want, but close enough that I don't care to finetune yet.

Every commit should be properly indented and not cause more warnings to appear!

April 27, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Moving closer to 2.4 stabilization (April 27, 2015, 17:18 UTC)

The SELinux userspace project has released version 2.4 in february this year, after release candidates have been tested for half a year. After its release, we at the Gentoo Hardened project have been working hard to integrate it within Gentoo. This effort has been made a bit more difficult due to the migration of the policy store from one location to another while at the same time switching to HLL- and CIL based builds.

Lately, 2.4 itself has been pretty stable, and we’re focusing on the proper migration from 2.3 to 2.4. The SELinux policy has been adjusted to allow the migrations to work, and a few final fixes are being tested so that we can safely transition our stable users from 2.3 to 2.4. Hopefully we’ll be able to stabilize the userspace this month or beginning of next month.

Sebastian Pipping a.k.a. sping (homepage, bugs)
Comment vulnerability in WordPress 4.2 (April 27, 2015, 13:21 UTC)

Hanno Böck tweeted about WordPress 4.2 Stored XSS rather recently. The short version is: if an attacker can comment on your blog, your blog is owned.

Since the latest release is affected and is the version I am using, I have been looking for a way to disable comments globally, at least until a fix has been released.
I’m surprised how difficult disabling comments globally is.

Option “Allow people to post comments on new articles” is filed under “Default article settings”, so it applies to new posts only. Let’s disable that.
There is a plug-in Disable comments, but since it claims to not alter the database (unless in persistent mode), I have a feeling that it may only remove commenting forms but leave commenting active to hand-made GET/POST requests, so that may not be safe.

So without studying WordPress code in depth my impression is that I have two options:

  • a) restrict comments to registered users, deactivate registration (hoping that all existing users are friendly and that disabled registration is waterproof in 4.2) and/or
  • b) disable comments for future posts in the settings (in case I post again before an update) and for every single post from the past.

On database level, the former can be seen here:

mysql> SELECT option_name, option_value FROM wp_options
           WHERE option_name LIKE '%regist%';
| option_name          | option_value |
| users_can_register   | 0            |
| comment_registration | 1            |
2 rows in set (0.01 sec)

For the latter, this is how to disable comments on all previous posts:

mysql> UPDATE wp_posts SET comment_status = 'closed';
Query OK, .... rows affected (.... sec)
Rows matched: ....  Changed: ....  Warnings: 0

If you have comments to share, please use e-mail this time. Upgraded to 4.2.1 now.

April 26, 2015
Hanno Böck a.k.a. hanno (homepage, bugs)

Intercepted certificate by AvastLately a lot of attention has been payed to software like Superfish and Privdog that intercepts TLS connections to be able to manipulate HTTPS traffic. These programs had severe (technically different) vulnerabilities that allowed attacks on HTTPS connections.

What these tools do is a widespread method. They install a root certificate into the user's browser and then they perform a so-called Man in the Middle attack. They present the user a certificate generated on the fly and manage the connection to HTTPS servers themselves. Superfish and Privdog did this in an obviously wrong way, Superfish by using the same root certificate on all installations and Privdog by just accepting every invalid certificate from web pages. What about other software that also does MitM interception of HTTPS traffic?

Antivirus software intercepts your HTTPS traffic

Many Antivirus applications and other security products use similar techniques to intercept HTTPS traffic. I had a closer look at three of them: Avast, Kaspersky and ESET. Avast enables TLS interception by default. By default Kaspersky intercepts connections to certain web pages (e. g. banking), there is an option to enable interception by default. In ESET TLS interception is generally disabled by default and can be enabled with an option.

When a security product intercepts HTTPS traffic it is itself responsible to create a TLS connection and check the certificate of a web page. It has to do what otherwise a browser would do. There has been a lot of debate and progress in the way TLS is done in the past years. A number of vulnerabilities in TLS (upon them BEAST, CRIME, Lucky Thirteen, FREAK and others) allowed to learn much more how to do TLS in a secure way. Also, problems with certificate authorities that issued malicious certificates (Diginotar, Comodo, Türktrust and others) led to the development of mitigation technologies like HTTP Public Key Pinning (HPKP) and Certificate Transparency to strengthen the security of Certificate Authorities. Modern browsers protect users much better from various threats than browsers used several years ago.

You may think: "Of course security products like Antivirus applications are fully aware of these developments and do TLS and certificate validation in the best way possible. After all security is their business, so they have to get it right." Unfortunately that's only what's happening in some fantasy IT security world that only exists in the minds of people that listened to industry PR too much. The real world is a bit different: All Antivirus applications I checked lower the security of TLS connections in one way or another.

Disabling of HTTP Public Key Pinning

Each and every TLS intercepting application I tested breaks HTTP Public Key Pinning (HPKP). It is a technology that a lot of people in the IT security community are pretty excited about: It allows a web page to pin public keys of certificates in a browser. On subsequent visits the browser will only accept certificates with these keys. It is a very effective protection against malicious or hacked certificate authorities issuing rogue certificates.

Browsers made a compromise when introducing HPKP. They won't enable the feature for manually installed certificates. The reason for that is simple (although I don't like it): If they hadn't done that they would've broken all TLS interception software like these Antivirus applications. But the applications could do the HPKP checking themselves. They just don't do it.

KasperskyKaspersky vulnerable to FREAK and CRIME

Having a look at Kaspersky, I saw that it is vulnerable to the FREAK attack, a vulnerability in several TLS libraries that was found recently. Even worse: It seems this issue has been reported publicly in the Kaspersky Forums more than a month ago and it is not fixed yet. Please remember: Kaspersky enables the HTTPS interception by default for sites it considers as especially sensitive, for example banking web pages. Doing that with a known security issue is extremely irresponsible.

I also found a number of other issues. ESET doesn't support TLS 1.2 and therefore uses a less secure encryption algorithm. Avast and ESET don't support OCSP stapling. Kaspersky enables the insecure TLS compression feature that will make a user vulnerable to the CRIME attack. Both Avast and Kaspersky accept nonsensical parameters for Diffie Hellman key exchanges with a size of 8 bit. Avast is especially interesting because it bundles the Google Chrome browser. It installs a browser with advanced HTTPS features and lowers its security right away.

These TLS features are all things that current versions of Chrome and Firefox get right. If you use them in combination with one of these Antivirus applications you lower the security of HTTPS connections.

There's one more interesting thing: Both Kaspersky and Avast don't intercept traffic when Extended Validation (EV) certificates are used. Extended Validation certificates are the ones that show you a green bar in the address line of the browser with the company name. The reason why they do so is obvious: Using the interception certificate would remove the green bar which users might notice and find worrying. The message the Antivirus companies are sending seems clear: If you want to deliver malware from a web page you should buy an Extended Validation certificate.

Everyone gets HTTPS interception wrong - just don't do it

So what do we make out of this? A lot of software products intercept HTTPS traffic (antiviruses, adware, youth protection filters, ...), many of them promise more security and everyone gets it wrong.

I think these technologies are a misguided approach. The problem is not that they make mistakes in implementing these technologies, I think the idea is wrong from the start. Man in the Middle used to be a description of an attack technique. It seems strange that it turned into something people consider a legitimate security technology. Filtering should happen on the endpoint or not at all. Browsers do a lot these days to make your HTTPS connections more secure. Please don't mess with that.

I question the value of Antivirus software in a very general sense, I think it's an approach that has very fundamental problems in itself and often causes more harm than good. But at the very least they should try not to harm other working security mechanisms.

(You may also want to read this EFF blog post: Dear Software Vendors: Please Stop Trying to Intercept Your Customers’ Encrypted Traffic)

Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
New devbox running (April 26, 2015, 16:31 UTC)

I announced it in February that Excelsior, which ran the Tinderbox, was no longer at Hurricane Electric. I have also said I'll start on working on a new generation Tinderbox, and to do that I need a new devbox, as the only three Gentoo systems I have at home are the laptops and my HTPC, not exactly hardware to run compilation all the freaking time.

So after thinking of options, I decided that it was much cheaper to just rent a single dedicated server, rather than a full cabinet, and after asking around for options I settled for, because of price and recommendation from friends. Unfortunately they do not support Gentoo as an operating system, which makes a few things a bit more complicated. They do provide you with a rescue system, based on Ubuntu, which is enough to do the install, but not everything is easy that way either.

Luckily, most of the configuration (but not all) was stored in Puppet — so I only had to rename the hosts there, changed the MAC addresses for the LAN and WAN interfaces (I use static naming of the interfaces as lan0 and wan0, which makes many other pieces of configuration much easier to deal with), changed the IP addresses, and so on. Unfortunately since I didn't start setting up that machine through Puppet, it also meant that it did not carry all the information to replicate the system, so it required some iteration and fixing of the configuration. This also means that the next move is going to be easier.

The biggest problem has been setting up correctly the MDRAID partitions, because of GRUB2: if you didn't know, grub2 has an automagic dependency on mdadm — if you don't install it it won't be able to install itself on a RAID device, even though it can detect it; the maintainer refused to add an USE flag for it, so you have to know about it.

Given what can and cannot be autodetected by the kernel, I had to fight a little more than usual and just gave up and rebuilt the two (/boot and / — yes laugh at me but when I installed Excelsior it was the only way to get GRUB2 not to throw up) arrays as metadata 0.90. But the problem was being able to tell what the boot up errors were, as I have no physical access to the device of course.

The server I rented is a Dell server, that comes with iDRAC for remote management (Dell's own name for IPMI, essentially), and allows you to set up connections to through your browser, which is pretty neat — they use a pool of temporary IP addresses and they only authorize your own IP address to connect to them. On the other hand, they do not change the default certificates, which means you end up with the same untrustable Dell certificate every time.

From the iDRAC console you can't do much, but you can start up the remove, JavaWS-based console, which reminded me of something. Unfortunately the JNLP file that you can download from iDRAC did not work on either Sun, Oracle or IcedTea JREs, segfaulting (no kidding) with an X.509 error log as last output — I seriously thought the problem was with the certificates until I decided to dig deeper and found this set of entries in the JNLP file:

 <resources os="Windows" arch="x86">
   <nativelib href="https://idracip/software/avctKVMIOWin32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLWin32.jar" download="eager"/>
 <resources os="Windows" arch="amd64">
   <nativelib href="https://idracip/software/avctKVMIOWin64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLWin64.jar" download="eager"/>
 <resources os="Windows" arch="x86_64">
   <nativelib href="https://idracip/software/avctKVMIOWin64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLWin64.jar" download="eager"/>
  <resources os="Linux" arch="x86">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  <resources os="Linux" arch="i386">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  <resources os="Linux" arch="i586">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  <resources os="Linux" arch="i686">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  <resources os="Linux" arch="amd64">
    <nativelib href="https://idracip/software/avctKVMIOLinux64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux64.jar" download="eager"/>
  <resources os="Linux" arch="x86_64">
    <nativelib href="https://idracip/software/avctKVMIOLinux64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux64.jar" download="eager"/>
  <resources os="Mac OS X" arch="x86_64">
    <nativelib href="https://idracip/software/avctKVMIOMac64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLMac64.jar" download="eager"/>

Turns out if you remove everything but the Linux/x86_64 option, it does fetch the right jar and execute the right code without segfaulting. Mysteries of Java Web Start I guess.

So after finally getting the system to boot, the next step is setting up networking — as I said I used Puppet to set up the addresses and everything, so I had working IPv4 at boot, but I had to fight a little longer to get IPv6 working. Indeed IPv6 configuration with servers, virtual and dedicated alike, is very much an unsolved problem. Not because there is no solution, but mostly because there are too many solutions — essentially every single hosting provider I ever used had a different way to set up IPv6 (including none at all in one case, so the only option was a tunnel) so it takes some fiddling around to set it up correctly.

To be honest, has a better set up than OVH or Hetzner, the latter being very flaky, and a more self-service one that Hurricane, which was very flexible, making it very easy to set up, but at the same time required me to just mail them if I wanted to make changes. They document for dibbler, as they rely on DHCPv6 with DUID for delegation — they give you a single /56 v6 net that you can then split up in subnets and delegate independently.

What DHCPv6 in this configuration does not give you is routing — which kinda make sense, as you can use RA (Route Advertisement) for it. Unfortunately at first I could not get it to work. Turns out that, since I use subnets for the containerized network, I enabled IPv6 forwarding, through Puppet of course. Turns out that Linux will ignore Route Advertisement packets when forwarding IPv6 unless you ask it nicely to — by setting accept_ra=2 as well. Yey!

Again this is the kind of problems that finding this information took much longer than it should have been; Linux does not really tell you that it's ignoring RA packets, and it is by far not obvious that setting one sysctl will disable another — unless you go and look for it.

Luckily this was the last problem I had, after that the server was set up fine and I just had to finish configuring the domain's zone file, and the reverse DNS and the SPF records… yes this is all the kind of trouble you go through if you don't just run your whole infrastructure, or use fully cloud — which is why I don't consider self-hosting a general solution.

What remained is just bits and pieces. The first was me realizing that Puppet does not remove the entries from /etc/fstab by default, so I noticed that the Gentoo default /etc/fstab file still contains the entries for CD-ROM drives as well as /dev/fd0. I don't remember which was the last computer with a floppy disk drive that I used, let alone owned.

The other fun bit has been setting up the containers themselves — similarly to the server itself, they are set up with Puppet. Since the server used to be running a tinderbox, it used to also host a proper rsync mirror, it was just easier, but I didn't want to repeat that here, and since I was unable to find a good mirror through mirrorselect (longer story), I configured Puppet to just provide to all the containers with as their sync server, which did not work. Turns out that our default mirror address does not have any IPv6 hosts on it ­– when I asked Robin about it, it seems like we just don't have any IPv6-hosted mirror that can handle that traffic, it is sad.

So anyway, I now have a new devbox and I'm trying to set up the rest of my repositories and access (I have not set up access to Gentoo's repositories yet which is kind of the point here.) Hopefully this will also lead to more technical blogging in the next few weeks as I'm cutting down on the overwork to relax a bit.

Sebastian Pipping a.k.a. sping (homepage, bugs)

I ran into an example of a web application vulnerable to Apache AddHandler/AddType misconfiguration by chance today.

The releases notes of Magento Community Edition 1.9.1 point to a remote code execution vulnerability.
Interestingly, the section Determining Your Vulnerability to the File System Attack is precisely a switch from AddHandler to SetHandler.

Fantastic! Let’s see if I can use that to convince web hoster X that is still arguing that use of AddHandler would be good enough.

PS: Before anyone takes this for advice to switch to vanilla, be sure to apply post-release easy-to-overlook patch “SUPEE-5344″, too. Details are up at (German) Magento-Shops stehen Angreifern offen or (English) Analyzing the Magento Vulnerability.

April 25, 2015
Git changes & impact to Overlays hostnames (April 25, 2015, 00:00 UTC)

As previously announced [1] [2], and previously in the discussion of merging Overlays with Gentoo’s primary SCM hosting (CVS+Git): The old overlays hostnames ( and have now been disabled, as well as non-SSH traffic to This was a deliberate move to seperate anonymous versus authenticated Git traffic, and ensure that anonymous Git traffic can continued to be scaled when we go ahead with switching away from CVS. Anonymous and authenticated Git is now served by seperate systems, and no anonymous Git traffic is permitted to the authenticated Git server.

If you have anonymous Git checkouts from any of the affected hostnames, you should switch them to using one of these new URLs:

  • git://$REPO

If you have authenticated Git checkouts from the same hosts, you should switch them to this new URL:

  • git+ssh://$REPO

In either case, you can trivially update any existing checkout with:
git remote set-url origin git+ssh://$REPO
(be sure to adjust the path of the repository and the name of the remote as needed).

April 23, 2015
Denis Dupeyron a.k.a. calchan (homepage, bugs)

is 46.

In a previous post I described how to patch QEMU to allow building binutils in a cross chroot. In there I increased the maximal number of argument pages to 64 because I was just after a quick fix. Today I finally bisected that, and the result is you need at least 46 for MAX_ARG_PAGES in order for binutils to build.

In bug 533882 it is discussed that LibreOffice requires an even larger number of pages. It is possible other packages also require such a large limit. Note that it may not be a good idea to increase the MAX_ARG_PAGES limit to an absurdly high number and leave it at that. A large amount of memory will be allocated in the target’s memory space and that may be a problem.

Hopefully QEMU switches to a dynamic limit someday like the kernel. In the meantime, my upcoming crossroot tool will offer a way to more easily deal with that.

April 21, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

The weekend of Saturday, April 11, my best friend and I made the drive down to Huntsville, Alabama to participate in the 2015 Superheroes Race benefiting the outstanding organisation that is The National Children’s Advocacy Center. It was a bit of a drive, but almost completely motorway, so it wasn’t that bad. Before talking about the main event (the race itself), I have to mention some of the way cool things that we did beforehand.

NASA US Space and Rocket Center in Huntsville, AL
NASA US Space and Rocket Center in Huntsville, AL
(click to enlarge)

Firstly, we went to the US Space and Rocket Center, which is essentially the visitor’s centre for NASA. In my opinion, the price of admission was a little high, but the exhibits made it completely worth it. Some of the highlights were the Robot Zoo and the amazing showroom of the Saturn V rocket.

Robot grasshopper at US Space and Rocket Center Robot Zoo   Saturn V rocket at US Space and Rocket Center
(click to enlarge)

Secondly, we had some great food at two very different types of restaurants. The first place was for a quick lunch at a Mexican place called Taqueria El Cazador #2, which is housed inside of a bus. We both had the tacos, which were excellent. I also grabbed some tamales, but they didn’t impress me nearly as much as the tacos. Overall, though, a great place that is cheap and tasty! The place that we went for dinner was Phuket Thai Restaurant, and even though it wasn’t in a bus, it was outstanding! Having been to Thailand, I was a little suspicious of the reviewers saying that it was one of the best places for Thai in the US. Having not eaten at all the Thai places in the US, I can’t say so definitively, but it was great! In particular, the Khao Soi was deliciously creamy, and putting the remaining sauce on some extra rice almost made for another meal in and of itself!

Thirdly, we went to the National Children’s Advocacy Center (NCAC) to see what needed to be done to prepare for the race the next day. I have been a supporter of the organisation for quite some time (you can see more about them via their link in the “Charities I Support” menu at the top of the blog), but I had never seen the facilities. I was stunned at the size of the complex, and even more taken aback at the people who work for the NCAC! They were absolutely outstanding, and after having seen the facilities, I feel even more strongly aligned with their mission statement of “modelling, promoting, and delivering excellence in child abuse response and prevention through service, education, and leadership.”

National Children's Advocacy Center NCAC sign   National Children's Advocacy Center NCAC community services building
(click to enlarge)

Fourthly, we just happened to stumble upon this really neat place known as Lowe Mill, and they were having an art show and free concert. There were various artists ranging from painters, photographers, and musicians, as well as plenty of bakers and confectioners serving up their yummy treats (like a decadent lemon cupcake with cream cheese frosting [I won't say who ate that one...]).

Okay, so now let’s get to the actual reason that I went to Huntsville: to support the NCAC in the 2015 Superheroes Race! There were three options for runners: 1) the timed and competitive 5K; 2) the timed and competitive 1-mile; 3) the 1-mile fun-run for kids (and adults too!). I participated in the 5K, and actually ended up coming in first place. My time wasn’t as good as I would have liked, but the course was a lot of fun! It went winding through the city centre and some very nice suburban areas as well (both starting and ending at the NCAC). There were also awards given out for the costumes (best individual costume, best family costumes, et cetera). I went as The Flash, but dang, it’s difficult to run in a full costume, so I applaud those people that did it! The Chief of Police actually ran in the race, and did so in uniform (belt and all)–now that’s supporting the community!

National Children's Advocacy Center NCAC kids' costumes   National Children's Advocacy Center NCAC family costumes
(click to enlarge)

One of the highlights of the whole trip was getting to meet some of the wonderful people that make the NCAC happen. I was fortunate enough to meet Catherine Courtney (the Director of Development) and Caroline Nelson (the Development Campaign Coordinator) (both pictured below), as well as Chris Newlin (the Executive Director). These three individuals are not only a few of the folks that help the NCAC function on a day-to-day basis, but they are truly inspirational people who have huge hearts for helping the children to whom the NCAC provides services–their passion to help kids is nearly tangible!

National Children's Advocacy Center NCAC Catherine Courtney and Zach   National Children's Advocacy Center NCAC Caroline Nelson and Zach
LEFT: The Flash (AKA me) and Catherine Courtney        RIGHT: Caroline Nelson
(click to enlarge)

Overall, the trip to Huntsville was both pleasant and productive. I hope that I will be able to participate in the races in the future, and that I will be able to work with the NCAC for years and years to come. If you agree with their Mission Statement, I urge you to become a member of the Protectors’ Circle and support the incredibly important work that they do.


Donnie Berkholz a.k.a. dberkholz (homepage, bugs)
How to give a great talk, the lazy way (April 21, 2015, 15:42 UTC)

presenter modeGot a talk coming up? Want it to go well? Here’s some starting points.

I give a lot of talks. Often I’m paid to give them, and I regularly get very high ratings or even awards. But every time I listen to people speaking in public for the first time, or maybe the first few times, I think of some very easy ways for them to vastly improve their talks.

Here, I wanted to share my top tips to make your life (and, selfishly, my life watching your talks) much better:

  1. Presenter mode is the greatest invention ever. Use it. If you ignore or forget everything else in this post, remember the rainbows and unicorns of presenter mode. This magical invention keeps the current slide showing on the projector while your laptop shows something different — the current slide, a small image of the next slide, and your slide notes. The last bit is the key. What I put on my notes is the main points of the current slide, followed by my transition to the next slide. Presentations look a lot more natural when you say the transition before you move to the next slide rather than after. More than anything else, presenter mode dramatically cut down on my prep time, because suddenly I no longer had to rehearse. I had seamless, invisible crib notes while I was up on stage.
  2. Plan your intro. Starting strong goes a long way, as it turns out that making a good first impression actually matters. It’s time very well spent to literally script your first few sentences. It helps you get the flow going and get comfortable, so you can really focus on what you’re saying instead of how nervous you are. Avoid jokes unless most of your friends think you’re funny almost all the time. (Hint: they don’t, and you aren’t.)
  3. No bullet points. Ever. (Unless you’re an expert, and you probably aren’t.) We’ve been trained by too many years of boring, sleep-inducing PowerPoint presentations that bullet points equal naptime. Remember presenter mode? Put the bullet points in the slide notes that only you see. If for some reason you think you’re the sole exception to this, at a minimum use visual advances/transitions. (And the only good transition is an instant appear. None of that fading crap.) That makes each point appear on-demand rather than all of them showing up at once.
  4. Avoid text-filled slides. When you put a bunch of text in slides, people inevitably read it. And they read it at a different pace than you’re reading it. Because you probably are reading it, which is incredibly boring to listen to. The two different paces mean they can’t really focus on either the words on the slide or the words coming out of your mouth, and your attendees consequently leave having learned less than either of those options alone would’ve left them with.
  5. Use lots of really large images. Each slide should be a single concept with very little text, and images are a good way to force yourself to do so. Unless there’s a very good reason, your images should be full-bleed. That means they go past the edges of the slide on all sides. My favorite place to find images is a Flickr advanced search for Creative Commons licenses. Google also has this capability within Search Tools. Sometimes images are code samples, and that’s fine as long as you remember to illustrate only one concept — highlight the important part.
  6. Look natural. Get out from behind the podium, so you don’t look like a statue or give the classic podium death-grip (one hand on each side). You’ll want to pick up a wireless slide advancer and make sure you have a wireless lavalier mic, so you can wander around the stage. Remember to work your way back regularly to check on your slide notes, unless you’re fortunate enough to have them on extra monitors around the stage. Talk to a few people in the audience beforehand, if possible, to get yourself comfortable and get a few anecdotes of why people are there and what their background is.
  7. Don’t go over time. You can go under, even a lot under, and that’s OK. One of the best talks I ever gave took 22 minutes of a 45-minute slot, and the rest filled up with Q&A. Nobody’s going to mind at all if you use up 30 minutes of that slot, but cutting into their bathroom or coffee break, on the other hand, is incredibly disrespectful to every attendee. This is what watches, and the timer in presenter mode, and clocks, are for. If you don’t have any of those, ask a friend or make a new friend in the front row.
  8. You’re the centerpiece. The slides are a prop. If people are always looking at the slides rather than you, chances are you’ve made a mistake. Remember, the focus should be on you, the speaker. If they’re only watching the slides, why didn’t you just post a link to Slideshare or Speakerdeck and call it a day?

I’ve given enough talks that I have a good feel on how long my slides will take, and I’m able to adjust on the fly. But if you aren’t sure of that, it might make sense to rehearse. I generally don’t rehearse, because after all, this is the lazy way.

If you can manage to do those 8 things, you’ve already come a long way. Good luck!

Tagged: communication, gentoo