Gentoo Logo
Gentoo Logo Side
Gentoo Spaceship

Contributors:
. Aaron W. Swenson
. Agostino Sarubbo
. Alec Warner
. Alex Alexander
. Alex Legler
. Alexey Shvetsov
. Alexis Ballier
. Alexys Jacob
. Amadeusz Żołnowski
. Andreas K. Hüttel
. Andreas Proschofsky
. Anthony Basile
. Arun Raghavan
. Bernard Cafarelli
. Bjarke Istrup Pedersen
. Brent Baude
. Brian Harring
. Christian Ruppert
. Chí-Thanh Christopher Nguyễn
. Daniel Gryniewicz
. David Abbott
. Denis Dupeyron
. Detlev Casanova
. Diego E. Pettenò
. Domen Kožar
. Donnie Berkholz
. Doug Goldstein
. Eray Aslan
. Fabio Erculiani
. Gentoo Haskell Herd
. Gentoo Monthly Newsletter
. Gentoo News
. Gilles Dartiguelongue
. Greg KH
. Hanno Böck
. Hans de Graaff
. Ian Whyman
. Ioannis Aslanidis
. Jan Kundrát
. Jason Donenfeld
. Jeffrey Gardner
. Jeremy Olexa
. Joachim Bartosik
. Johannes Huber
. Jonathan Callen
. Jorge Manuel B. S. Vicetto
. Joseph Jezak
. Kenneth Prugh
. Lance Albertson
. Liam McLoughlin
. LinuxCrazy Podcasts
. Luca Barbato
. Luis Francisco Araujo
. Mark Loeser
. Markos Chandras
. Mart Raudsepp
. Matt Turner
. Matthew Marlowe
. Matthew Thode
. Matti Bickel
. Michael Palimaka
. Michal Hrusecky
. Michał Górny
. Mike Doty
. Mike Gilbert
. Mike Pagano
. Nathan Zachary
. Ned Ludd
. Nirbheek Chauhan
. Pacho Ramos
. Patrick Kursawe
. Patrick Lauer
. Patrick McLean
. Pavlos Ratis
. Paweł Hajdan, Jr.
. Petteri Räty
. Piotr Jaroszyński
. Rafael Goncalves Martins
. Raúl Porcel
. Remi Cardona
. Richard Freeman
. Robin Johnson
. Ryan Hill
. Sean Amoss
. Sebastian Pipping
. Steev Klimaszewski
. Stratos Psomadakis
. Sune Kloppenborg Jeppesen
. Sven Vermeulen
. Sven Wegener
. Thomas Kahle
. Tiziano Müller
. Tobias Heinlein
. Tobias Klausmann
. Tom Wijsman
. Tomáš Chvátal
. Vikraman Choudhury
. Zack Medico

Last updated:
September 16, 2014, 13:04 UTC

Disclaimer:
Views expressed in the content published here do not necessarily represent the views of Gentoo Linux or the Gentoo Foundation.


Bugs? Comments? Suggestions? Contact us!

Powered by:
Planet Venus

Welcome to Gentoo Universe, an aggregation of weblog articles on all topics written by Gentoo developers. For a more refined aggregation of Gentoo-related topics only, you might be interested in Planet Gentoo.

September 15, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
My thoughts on the Self-Hosting Solution (September 15, 2014, 19:38 UTC)

You probably noticed that in the (frequent) posts talking about security and passwords lately, I keep suggesting LastPass as a password manager. This is the manager that I use myself, and the reason why I came to this one is multi-faceted, but essentially I'm suggesting you use a tool that does not make it more inconvenient to maintain proper password hygiene. Because yes, you should be using different passwords, with a combination of letters, numbers and symbols, but if you have to come up with a new one every time, then things are going to be difficult and you'll just decide to use the same password over and over.

Or you'll use a method for having "unique" passwords that are actually comprised of a fixed part and a mobile one (which is what I used for the longest time). And let's be clear, using the same base password suffixed with the name of the site you're signing up for is not a protection at all, the moment more than one of your passwords is discovered.

So convenience being important, because inconvenience just leads to bad security hygiene, LastPass delivers on what I need: it has autofill, so I don't have to open a terminal and run sgeps (like I used to be) to get the password out of the store, it generates the password in the browser, so I don't have to open a terminal and run pwgen, it runs on my cellphone, so I can use it to fetch the password to type somewhere else, and it even auto-fills my passwords in the Android apps, so I don't have to use a simple password when dealing with some random website that then patches to an app on my phone. But it also has a few good "security conveniences": you can re-encode your Vault on a new master password, you can use a proper OTP pad or a 2FA device to protect it further, and they have some extras such as letting you know if the email you use across services are involved in an account breach.

This does not mean there are no other good password management tools, I know the name of plenty, but I just looked for one that had the features I cared about, and I went with it. I'm happy with LastPass right now. Yes, I need to trust the company and their code a fair bit, but I don't think that just being open source would gain me more trust. Being open source and audited for a long time, sure, but I don't think either way it's a dealbreaker for me. I mean Chrome itself has a password manager, it just feels not suitable for me (no generation, no obvious way to inspect the data from mobile, sometimes bad collation of URLs, and as far as I know no way to change the sync encryption password). It also requires me to have access to my Google account to get that data.

But the interesting part is how geeks will quickly suggest to just roll your own, be it using some open-source password manager, requiring an external sync system (I did that for sgeps, but it's tied to a single GPG key, so it's not easy for me having two different hardware smartcards), or even your own sync infrastructure. And this is what I really can't stand as an answer, because it solves absolutely nothing. Jürgen called it cynical last year, but I think it's even worse than that, it's hypocritical.

Roll-your-own or host-your-own are, first of all, not going to be options for the people who have no intention to learn how computer systems work — and I can't blame them, I don't want to know how my fridge or dishwasher work, I just want them working. People don't care to learn that you can get file A on computer B, but then if you change it on both while offline you'll have collisions, so now you lost one of the two changes. They either have no time, or just no interest or (but I don't think that happens often) no skill to understand that. And it's not just the random young adult that ends up registering on xtube because they have no idea what it means. Jeremy Clarkson had to learn the hard way what it means to publish your bank details to the world.

But I think it's more important to think of the amount of people who think that they have the skills and the time, and then are found lacking one or both of them. Who do you think can protect your content (and passwords) better? A big company with entire teams dedicated to security, or an average 16 years old guy who think he can run the website's forum? — The reference here is to myself: back in 2000/2001 I used to be the forum admin for an Italian gaming community. We got hacked, multiple times, and every time it was for me a new discovery of what security is. At the time third-party forum hosting was reserved to paying customers, and the results have probably been terrible. My personal admin password matched one of my email addresses up until last week and I know for a fact that at least one group of people got access to the password database, where they were stored in plain text.

Yes it is true, targets such as Adobe will lead to many more valid email addresses and password hashes than your average forum, but as the "fake" 5M accounts should have shown you, targeting enough small fishes can lead to just about the same results, if not even better, as you may be lucky and stumble across two passwords for the same account, which allows you to overcome the above-mentioned similar-but-different passwords strategy. Indeed, as I noted in my previous post, Comic Book Database admitted to be the source of part of that dump, and it lists at least four thousand public users (contributors). Other sites such as MythTV Talk or PoliceAuctions.com, both also involved, have no such statement ether.

This is not just a matter of the security of the code itself, so the "many eyes" answer does not apply. It is very well possible to have a screw up with an open source program as well, if it's misconfigured, or if a vulnerable version don't get updated in time because the admin just has no time. You see that all the time with WordPress and its security issues. Indeed, the reason why I don't migrate my blog to WordPress is that I won't ever have enough time for it.

I have seen people, geeks and non-geeks both, taking the easy way out too many times, blaming Apple for the nude celebrity pictures or Google for the five million accounts. It's a safe story: "the big guys don't know better", "you just should keep it off the Internet", "you should run your own!" At the end of the day, both turned out to be collections, assembled from many small cuts, either targeted or not, in part due to people's bad password hygiene (or operational security if you prefer a more geek term), and in part due to the fact that nothing is perfect.

September 13, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Make password management better (September 13, 2014, 20:13 UTC)

When I posted my previous post on accounts on Google+ I received a very interesting suggestions that I would like to bring to the attention of more people. Andrew Cooks pointed out that what LastPass (and other password managers) really need, is a way to specify the password policy programmatically, rather than crowdsourcing this data as LastPass is doing right now.

There are already a number of cross-product specifications of fixed-path files used to describe parameters such as robots.txt or sitemap.xml. While cleaning up my blog's image serving I also found that there is a rules.abe file used by the NoScript extension for Firefox. In this optic, adding a new password-policy.txt file to define some parameters for the password policy of the website.

Things like the minimum and maximum length of the password, which characters are allowed, whether it is case sensitive or not. These are important information that all the password managers need to know, and as I said not all websites make it clear to the user either. I'll recount two different horror stories, one in the past and one more recent, that show how that is important.

The first story is from probably almost ten years ago or so. I registered with the Italian postal service. I selected a "strong" (not really) password, 11 characters long. It was not really dictionary-based, but it was close enough if you knew my passwords' pattern. Anyway, I liked the idea of having the long password. I signed up for it, I logged back in, everything worked. Until a few months later, when I decided I wanted to fetch that particular mailbox from GMail — yes, the Italian postal service gives you an email box, no I don't want to comment further on that.

What happens is that the moment I tried to set up the mail fetching on GMail, it kept failing authentication. And I'm sure I used the right password that I've insisted using up to that point! I log in on the website just fine with it, so what gives? A quick check at the password that my browser (I think Firefox at the time) think is the password of that website shows me the truth: the password I've been using to log in does not match the one I tried to use from GMail: the last character is not there. Some further inspection of the postal service website shows that the password fields, both in the password change and login (and I assumed at the time the registration page for obvious reasons), set a maxlength value to 10. So of course, as long as I typed or pasted the password in the field, the way I typed it when I registered, it worked perfectly fine, but when I tried to login out of band (through POP3) it used the password as I intended, and failed.

A similar, more recent story happened with LastMinute. I went to change my password, in my recent spree of updating all my passwords, even for accounts not in use (mostly to make sure that they don't get leaked and allow people to do crazy stuff to me). My default password generator on LastPass is set to generate 32-characters passwords. But that did not work for LastMinute, or rather, it appeared to. It let me change my password just fine, but when I tried logging back in, well, it did not work. Yes, this is the reason that I try to log back in after generating the password, I've seen that happening before. In this case, the problem was to be found in the length of the password.

But just having a proper range for the password length wouldn't be enough. Other details that would be useful are for instance the allowed symbols; I have found that sometimes I need to either generate a number of passwords to find one that does not have one of the disallowed symbols but still has some, or give up on the symbols altogether and ask LastPass to generate only letters and numbers. Or having a way to tell that the password is case sensitive or not — because if it is not, what I do is disable the generation of one set of letters, so that it randomises them better.

But there is more metadata that could be of use there — things like which domains should the password be used with, for instance. Right now LastPass has a (limited) predefined list of equivalent domains, and hostnames that need to match exactly (so that bugs.gentoo.org and forums.gentoo.org are given different passwords), while it's up to you to build the rest of the database. Even for the Amazon domains, the list is not comprehensive and I had to add quite a few when logging in the Italian and UK stores.

Of course if you were just to tell that your website uses the same password as, say, google.com, you're going to have a problem. What you need is a reciprocal indication that both sites think the other is equivalent, basically serving the same identical file. This makes the implementation a bit more complex but it should not be too difficult as those kind of sites tend to at least share robots.txt (okay, not in the case of Amazon), so distributing one more file should not be that difficult.

I'm not sure if anybody is going to work on implementing this, or writing a proper specification for it, rather than a vague rant on a blog, but hope can't die, right?

Sebastian Pipping a.k.a. sping (homepage, bugs)
My first cover on a printed book (September 13, 2014, 14:03 UTC)

A few days ago I had the chance to first get my hands on a printed version of that book I designed the cover for: Einführung in die Mittelspieltaktik des Xiangqi by Rainer Schmidt. The design was done using Inkscape and xiangqi-setup. I helped out with a few things on the inside too.

A few links on the actual book:

PS: Please note the cover images are “all rights reserved”.

Gentoo Haskell Herd a.k.a. haskell (homepage, bugs)
ghc 7.8.3 and rare architectures (September 13, 2014, 09:03 UTC)

After some initially positive experience with ghc-7.8-rc1 I’ve decided to upstream most of gentoo fixes.

On rare arches ghc-7.8.3 behaves a bit bad:

  • ia64 build stopped being able to link itself after ghc-7.4 (gprel overflow)
  • on sparc, ia64 and ppc ghc was not able to create working shared libraries
  • integer-gmp library on ia64 crashed, and we had to use integer-simple

I have written a small story of those fixes here if you are curious.

TL;DR:

To get ghc-7.8.3 working nicer for exotic arches you will need to backport at least the following patches:

Thank you!


September 10, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
How to analyze a dump of usernames (September 10, 2014, 14:29 UTC)

There has been some noise around a leak of users/passwords pairs which somehow panicked people into thinking it was coming from a particular provider. Since it seems most people have not even tried looking at the account information available, I'd like to point out some ways that could have helped avoiding the panic, if only the reporters cared. It also fits nicely into my previous notes on accounts' churn.

But before proceeding let me make one thing straight: this post contains no information that is not available to the public and bears no relation to my daily work for my employer. Just wanted to make that clear. Edit: for the official response, please see this blog post of Google's Security blog.

To begin the analysis you need a copy of the list of usernames; Italian blogger Paolo Attivissimo linked to it in his post but I'm not going to do so. Especially since it's likely to become obsolete soon, and might not be liked by many. The archive is a compressed list of usernames without passwords or password hashes. At first, it seems to contain almost exclusively gmail.com addresses — in truth there are more addresses but it probably does not hit the news as much to say that there are some 5 million addresses from some thousand domains.

Let's first try to extract real email addresses from the file, which I'll call rawsource.txt — yes it does not match the name of the actual source file out there but I would rather avoid the search requests finding this post from the filename.

$ fgrep @ rawsource.txt > source-addresses.txt

This removes some two thousands lines that were not valid addresses — turns out that the file actually contains some passwords, so let's process it a little more to get a bigger sample of valid addresses:

$ sed -i -e 's:|.*::' source-addresses.txt

This should make the next command give us a better estimate of the content:

$ sed -n -e 's:.*@::p' source-addresses.txt | sort | uniq -c | sort -n
[snip]
  238 gmail.com.au
256 gmail.com.br
338 gmail.com.vn
608 gmail.com777
123215 yandex.ru
4800129 gmail.com

So as we were saying earlier there are more than just Google accounts in this. A good chunk of them are on Yandex, but if you look at the outlier in the list there are plenty of other domains including Yahoo. Let's just filter away the four thousands addresses using either broken domains or outlier domains and instead focus on these three providers:

$ egrep '@(gmail.com|yahoo.com|yandex.ru)$' source-addresses.txt > good-addresses.txt

Now things get more interesting, because to proceed to the next step you have to know how email servers and services work. For these three providers, and many default setups for postfix and similar, the local part of the address (everything before the @ sign) can contain a + sign, when that is found, the local part is split into user and extension, so that mail to nospam+noreally would be sent to the user nospam. Servers generally ignore the extension altogether, but you can use it to either register multiple accounts on the same mailbox (like I do for PayPal, IKEA, Sony, …) or to filter the received mail on different folders. I know some people who think they can absolutely identify the source of spam this way — I'm a bit more skeptical, if I was a spammer I would be dropping the extension altogether. Only some very die-hard Unix fans would not allow inbound email without an extension. Especially since I know plenty of services that don't accept email addresses with + in them.

Since this is not very well known, there are going to be very few email addresses using this feature, but that's still good because it limits the amount of data to crawl through. Finding a pattern within 5M addresses is going to take a while, finding one in 4k is much easier:

$ egrep '.*\+.*@.*' good-addresses.txt | sed -e '/.*@.*@.*/d' > experts-addresses.txt

The second command filters out some false positives due to two addresses being on the same line; the results from the source file I started with is 3964 addresses. Now we're talking. Let's extract the extensions from those good addresses:

$ sed -e 's:.*+\(.*\)@.*:\1:' experts-addresses.txt | sort > extensions.txt

The first obvious thing you can do is figure out if there are duplicates. While the single extensions are probably interesting too, finding a pattern is easier if you have people using the same extension, especially since there aren't that many. So let's see which extensions are common:

$ sed -e 's:.*+\(.*\)@.*:\1:' experts-addresses.txt | sort | uniq -c -d | sort -n > common-extensions.txt

An obvious quick look look of that shows that a good chunk of the extensions (the last line in the generated file) used were referencing xtube – which you may or may not know as a porn website – reminding me of the YouPorn-related leak two and a half years ago. Scouring through the list of extensions, it's also easy to spot the words "porn" and "porno", and even "nudeceleb" making the list probably quite up to date.

Just looking at the list of extensions shows a few patterns. Things like friendster, comicbookdb (and variants like comics, comicdb, …) and then daz (dazstudio), and mythtv. As RT points out it might very well be phishing attempts, but it is also well possible that some of those smaller sites such as comicbookdb were breached and people just used the same passwords for their GMail address as the services (I used to, too!), which is why I think mandatory registrations are evil.

The final automatic interesting discovery you can make involves checking for full domains in the extensions themselves:

fgrep . extensions.txt | sort -u

This will give you which extensions include a dot in the name, many of which are actually proper site domains: xtube figures again, and so does comicbookdb, friendster, mythtvtalk, dax3d, s2games, policeauctions, itickets and many others.

What does this all tell me? I think what happens is that this list was compiled with breaches of different small websites that wouldn't make a headline (and that most likely did not report to their users!), plus some general phishing. Lots of the passwords that have been confirmed as valid most likely come from people not using different passwords across websites. This breach is fixed like every other before it: stop using the same password across different websites, start using something like LastPass, and use 2 Factor Authentication everywhere is possible.

September 09, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Ramblings on audiobooks (September 09, 2014, 18:26 UTC)

In one of my previous posts I have noted I'm an avid audiobook consumer. I started when I was at the hospital, because I didn't have the energy to read — and most likely, because of the blood sugar being out of control after coming back from the ICU: it turns out that blood sugar changes can make your eyesight go crazy; at some point I had to buy a pair of €20 glasses simply because my doctor prescribed me a new treatment and my eyesight ricocheted out of control for a week or so.

Nowadays, I have trouble sleeping if I'm not listening to something, and I end up with the Audible app installed in all my phones and tablets, with at least a few books preloaded whenever I travel. Of course as I said, I keep the majority of my audiobooks in the iPod, and the reason is that while most of my library is on Audible, not all of it is. There are a few books that I have bought on iTunes before finding out about Audible, and then there are a few I received in CD form, including The Hitchhiker's Guide To The Galaxy Complete Radio Series which is my among my favourite playlists.

Unfortunately, to be able to convert these from CD to a format that the iPod could digest, I ended up having to buy a software called Audiobook Builder for Mac, which allows you to rip CDs and build M4B files out of them. What's M4B? It's the usual mp4 format container, just with an extension that makes iTunes consider it an audiobook, and with chapter markings in the stream. At the time I first ripped my audiobooks, ffmpeg/libav had no support for chapter markings, so that was not an option. I've been told that said support is there now, but I have not tried getting it to work.

Indeed, what I need to find out is how to build an audiobook file out of a string of mp3 files, and I have no idea how to fix that now that I no longer have access to my personal iTunes account on a mac to re-download the Audiobook Builder and process them. In particular, the list of mp3s that I'm looking forward to merge together are the years 2013 and 2014 of BBC's The News Quiz, to which I'm addicted and listen continuously. Being able to join them all together so I can listen to them with a multi-day-running playlist is one of the very few things that still let me sleep relatively calmly — I say relatively because I really don't remember when was the last time I have slept soundly in about an year by now.

Essentially, what I'd like is for Audible to let me sideload some content (the few books I did not buy from them, and the News Quiz series that I stitch together from the podcast), and create a playlist — then for what I'm concerned I don't have to use an iPod at all. Well, beside the fact that I'd have to find a way to shut up notifications while playing audiobooks. Having Dragons of Autumn Twilight interrupted by the Facebook pop notification is not something that I'm looking forward for most of the time. And in some cases I even have had some background update disrupting my playback so there is definitely space for improvement.

September 08, 2014
Gentoo Monthly Newsletter: August 2014 (September 08, 2014, 21:20 UTC)

Gentoo News

Council News

Concerning the handling of bash-completion and of phase functions in eclasses in general the council decided no actions. The former should be handled by the shell-tools team, the latter needs more discussion on the mailing lists.

Then we had two hot topics. The first was the games team policy; the council clarified that the games team has in no way authority over game ebuilds maintained by other developers. In addition, the games team should elect a lead in the near future. If it doesn’t it will be considered dysfunctional.  Tim Harder (radhermit) acts as interim lead and organizes the elections.

Next, rumors about the handling of dynamic dependencies in Portage had sparked quite a stir. The council asks the Portage team basically not to remove dynamic dependency handling before they haven’t worked out and presented a good plan how Gentoo would work without them. Portage tree policies and the
handling of eclasses and virtuals in particular need to be clarified.

Finally the list of planned features for EAPI 6 was amended by two items, namely additional options for configure and a non-runtime switchable ||= () or-dependency.

Gentoo Developer Moves

Summary

Gentoo is made up of 242 active developers, of which 43 are currently away.
Gentoo has recruited a total of 803 developers since its inception.

Changes

  • Ian Stakenvicius (axs) joined the multilib project
  • Michał Górny (mgorny) joined the QA team
  • Kristian Fiskerstrand (k_f) joined the Security team
  • Richard Freeman (rich0) joined the systemd team
  • Pavlos Ratis (dastergon) joined the Gentoo Infrastructure team
  • Patrice Clement (monsieur) and Ian Stakenvicius (axs) joined the perl team
  • Chris Reffett (creffett) joined the Wiki team
  • Pavlos Ratis (dastergon) left the KDE project
  • Dirkjan Ochtman (djc) left the ComRel project

Portage

This section summarizes the current state of the portage tree.

Architectures 45
Categories 162
Packages 17653
Ebuilds 37397
Architecture Stable Testing Total % of Packages
alpha 3661 574 4235 23.99%
amd64 10895 6263 17158 97.20%
amd64-fbsd 0 1573 1573 8.91%
arm 2692 1755 4447 25.19%
arm64 570 32 602 3.41%
hppa 3073 496 3569 20.22%
ia64 3196 626 3822 21.65%
m68k 614 98 712 4.03%
mips 0 2410 2410 13.65%
ppc 6841 2475 9316 52.77%
ppc64 4332 971 5303 30.04%
s390 1464 349 1813 10.27%
sh 1650 427 2077 11.77%
sparc 4135 922 5057 28.65%
sparc-fbsd 0 317 317 1.80%
x86 11572 5297 16869 95.56%
x86-fbsd 0 3241 3241 18.36%

gmn-portage-stats-2014-09

Security

The following GLSAs have been released by the Security Team

GLSA Package Description Bug
201408-19 app-office/openoffice-bin (and 3 more) OpenOffice, LibreOffice: Multiple vulnerabilities 283370
201408-18 net-analyzer/nrpe NRPE: Multiple Vulnerabilities 397603
201408-17 app-emulation/qemu QEMU: Multiple vulnerabilities 486352
201408-16 www-client/chromium Chromium: Multiple vulnerabilities 504328
201408-15 dev-db/postgresql-server PostgreSQL: Multiple vulnerabilities 456080
201408-14 net-misc/stunnel stunnel: Information disclosure 503506
201408-13 dev-python/jinja Jinja2: Multiple vulnerabilities 497690
201408-12 www-servers/apache Apache HTTP Server: Multiple vulnerabilities 504990
201408-11 dev-lang/php PHP: Multiple vulnerabilities 459904
201408-10 dev-libs/libgcrypt Libgcrypt: Side-channel attack 519396
201408-09 dev-libs/libtasn1 GNU Libtasn1: Multiple vulnerabilities 511536
201408-08 sys-apps/file file: Denial of Service 505534
201408-07 media-libs/libmodplug ModPlug XMMS Plugin: Multiple vulnerabilities 480388
201408-06 media-libs/libpng libpng: Multiple vulnerabilities 503014
201408-05 www-plugins/adobe-flash Adobe Flash Player: Multiple vulnerabilities 519790
201408-04 dev-util/catfish Catfish: Multiple Vulnerabilities 502536
201408-03 net-libs/libssh LibSSH: Information disclosure 503504
201408-02 media-libs/freetype FreeType: Arbitrary code execution 504088
201408-01 dev-php/ZendFramework Zend Framework: SQL injection 369139

Package Removals/Additions

Removals

Package Developer Date
virtual/perl-Class-ISA dilfridge 02 Aug 2014
virtual/perl-Filter dilfridge 02 Aug 2014
dev-vcs/gitosis robbat2 04 Aug 2014
dev-vcs/gitosis-gentoo robbat2 04 Aug 2014
virtual/python-argparse mgorny 11 Aug 2014
virtual/python-unittest2 mgorny 11 Aug 2014
app-emacs/sawfish ulm 19 Aug 2014
virtual/ruby-test-unit graaff 20 Aug 2014
games-action/d2x mr_bones_ 25 Aug 2014
games-arcade/koules mr_bones_ 25 Aug 2014
dev-lang/libcilkrts ottxor 26 Aug 2014

Additions

Package Developer Date
dev-python/oslotest prometheanfire 01 Aug 2014
dev-db/tokumx chainsaw 01 Aug 2014
sys-boot/gummiboot mgorny 02 Aug 2014
app-admin/supernova alunduil 03 Aug 2014
dev-db/mysql-cluster robbat2 03 Aug 2014
net-libs/txtorcon mrueg 04 Aug 2014
dev-ruby/prawn-table mrueg 06 Aug 2014
sys-apps/cv zx2c4 06 Aug 2014
media-libs/openctm amynka 07 Aug 2014
sci-libs/levmar amynka 07 Aug 2014
media-gfx/printrun amynka 07 Aug 2014
dev-python/alabaster idella4 10 Aug 2014
dev-haskell/regex-pcre slyfox 11 Aug 2014
dev-python/gcs-oauth2-boto-plugin vapier 12 Aug 2014
dev-python/astropy-helpers jlec 12 Aug 2014
dev-perl/Math-ModInt chainsaw 13 Aug 2014
dev-ruby/classifier-reborn mrueg 13 Aug 2014
media-gfx/meshlab amynka 14 Aug 2014
dev-libs/librevenge scarabeus 15 Aug 2014
www-apps/jekyll-coffeescript mrueg 15 Aug 2014
www-apps/jekyll-gist mrueg 15 Aug 2014
www-apps/jekyll-paginate mrueg 15 Aug 2014
www-apps/jekyll-watch mrueg 15 Aug 2014
sec-policy/selinux-salt swift 15 Aug 2014
www-apps/jekyll-sass-converter mrueg 15 Aug 2014
dev-ruby/rouge mrueg 15 Aug 2014
dev-ruby/ruby-beautify graaff 16 Aug 2014
sys-firmware/nvidia-firmware idl0r 17 Aug 2014
media-libs/libmpris2client ssuominen 20 Aug 2014
xfce-extra/xfdashboard ssuominen 20 Aug 2014
www-client/opera-developer jer 20 Aug 2014
dev-libs/openspecfun patrick 21 Aug 2014
dev-libs/marisa dlan 22 Aug 2014
media-sound/dcaenc beandog 22 Aug 2014
sci-mathematics/geogebra amynka 23 Aug 2014
dev-python/crumbs alunduil 25 Aug 2014
media-gfx/kxstitch kensington 26 Aug 2014
media-gfx/symboleditor kensington 26 Aug 2014
dev-perl/Sort-Key chainsaw 26 Aug 2014
dev-perl/Sort-Key-IPv4 chainsaw 26 Aug 2014
sci-visualization/yt xarthisius 26 Aug 2014
dev-ruby/globalid graaff 27 Aug 2014
dev-python/certifi idella4 27 Aug 2014
www-apps/jekyll-sitemap mrueg 27 Aug 2014
sys-apps/tuned dlan 29 Aug 2014
app-portage/g-sorcery jauhien 29 Aug 2014
app-portage/gs-elpa jauhien 29 Aug 2014
app-portage/gs-pypi jauhien 29 Aug 2014
app-admin/eselect-rust jauhien 29 Aug 2014
sys-block/raid-check chutzpah 29 Aug 2014
dev-python/python3-openid maksbotan 30 Aug 2014
dev-python/python-social-auth maksbotan 30 Aug 2014
dev-python/websocket-client alunduil 31 Aug 2014
dev-ruby/ethon graaff 31 Aug 2014

Bugzilla

The Gentoo community uses Bugzilla to record and track bugs, notifications, suggestions and other interactions with the development team.

Activity

The following tables and charts summarize the activity on Bugzilla between 01 August 2014 and 31 August 2014. Not fixed means bugs that were resolved as NEEDINFO, WONTFIX, CANTFIX, INVALID or UPSTREAM.
gmn-activity-2014-08

Bug Activity Number
New 1575
Closed 981
Not fixed 187
Duplicates 145
Total 6023
Blocker 5
Critical 19
Major 66

Closed bug ranking

The following table outlines the teams and developers with the most bugs resolved during this period

Rank Team/Developer Bug Count
1 Gentoo Security 102
2 Gentoo's Team for Core System packages 39
3 Gentoo KDE team 37
4 Default Assignee for Orphaned Packages 32
5 Julian Ospald (hasufell) 26
6 Gentoo Games 25
7 Portage team 25
8 Netmon Herd 24
9 Python Gentoo Team 23
10 Others 647

gmn-closed-2014-08

Assigned bug ranking

The developers and teams who have been assigned the most bugs during this period are as follows.

Rank Team/Developer Bug Count
1 Gentoo Linux bug wranglers 160
2 Gentoo Security 61
3 Default Assignee for Orphaned Packages 60
4 Gentoo KDE team 45
5 Gentoo's Team for Core System packages 45
6 Gentoo Linux Gnome Desktop Team 37
7 Gentoo Games 28
8 Portage team 28
9 Python Gentoo Team 26
10 Others 1084

gmn-opened-2014-08

Heard in the community

Send us your favorite Gentoo script or tip at gmn@gentoo.org

Getting Involved?

Interested in helping out? The GMN relies on volunteers and members of the community for content every month. If you are interested in writing for the GMN or thinking of another way to contribute, please send an e-mail to gmn@gentoo.org.

Comments or Suggestions?

Please head over to this forum post.

September 07, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Account churn (September 07, 2014, 14:30 UTC)

In my latest post I singled out ne of the worst security experiences I’ve had with a service, but that is by far not the only bad experience I had. Indeed, given that I’ve been actively hunting down my old accounts and tried to get my hands on them, I can tell you that I have plenty of material to fill books with bad examples.

First of all, there is the problem of migrating the email addresses. For the longest time I’ve been using my GMail address to register everywhere, but then I decided to migrate to use my own domain (especially since Gandi supports two factor authentication, which makes it much safer). Unfortunately that means that not only I have a bunch of accounts still on the old email address, but I also have duplicate accounts.

Duplicate accounts become even more tricky when you consider that I had my own company, which meant I had double accounts for things that did not allow me to ship sometimes with a VAT ID attached, and sometimes not. Sometimes I could close the accounts (once I dropped the VAT ID), and sometimes I couldn’t, so a good deal of them are still out there.

FInally, there are the services that are available in multiple countries but with country-specific accounts. Which, don’t be mistaken, does not mean that every country has its own account database! It simply means that a given account is assigned to a country and does not work on any other. In most cases you cannot even migrate your account across countries. This is the case, for instance, of OVH, and why I moved to Gandi but also of PayPal (in which the billing address is tied to the country of the account and can’t be changed), IKEA and -PSN- Sony Online Entertainment. The end result is that I have duplicated (or even triplicated) accounts to cover the fact I have lived in multiple countries by now.

Also, it turns out that I completely forgot how many services I registered to over the years. Yes I have the passwords as stored by Chrome, but that’s not a comprehensive list as some of the most critical passwords have never been saved there (such as my bank’s password), plus some websites I have never used in Chrome, and at some point I had it clean the history of passwords and start from scratch. Some of the passwords have been saved in sgeps so I could look them up there, but even those are not a complete list. I ended up looking in my old email content to figure out which accounts I forgot having. The results have been fun.

But what about the grievances? Some of the accounts I wanted to gain access to again ended up being blocked or deleted, I’m surprised by the amount of services that either were killed, or moved around. At least three ebook stores I used are now gone, two of which absorbed by Kobo, while Marks & Spencer refused to recognize my email as valid, I assume they at some point reset their user database or something. Some of the hotel loyalty programs I signed up before and used once or twice disappeared, or were renamed/merged into something else. Not a huge deal but it makes account management a fun problem.

Then there are the accounts that got their password invalidated in the mean time, so even if I have a copy of it, it’s useless. Given that some accounts I had not logged into for years, that’s fair to happen: between leaks, heartbleed, and the overdue changes in best practices for password storage, I am more bothered by the services that did not invalidate my password in the mean time. But then again, there are different ways to deal with it. Some services when trying to login with the previous password point out that it’s no long valid and proceed with the same Forgotten password request workflow. Others will send you the password by email in plain text.

One quite egregious case happened with an Italian electronics shop, one of the first online-only stores I know of in Italy. Somehow, I knew that the account was still active, mostly because I could see their newsletter in the spam folder of my GMail account. So I went and asked for the password back, to change the address and stop the newsletter too (given I don’t live in Italy any longer), they sent me back the userid and password in cleartext. They reset their passwords in the mean time, and the default password became my Italian tax ID. Not very safe, if I were to know the user id of anyone else, knowing their tax ID is easy, as it can be calculated based on a few personal, but not so secret, details (full name, sex, date and city of birth).

But there is something worse than unannounced password resets. The dance of generating a new password. I have now the impression that it’s a minority of services that actually allow you to use whichever password you want. Lots of the services I have changed password for between last night and today required me to disable the non-alphanumeric symbols, because either they don’t support any non-alphanumeric character, or they only support a subset that LastPass does not let you select.

But this is not as bothersome as the length limitation of passwords. Most sites will be happy to tell you that they require a minimum of 6 or 8 characters for your password — few will tell you upfront the maximum length of a password. And very few of those that won’t tell you right away will fix the mistake by telling you when the password is too long, how long it can be. I even found sites that simply error out on you when you try to use a password that is not valid, and decide to invalidate both your old and temporary passwords, while not accepting the new one. It’s a serious pain.

Finally, I’ve enabled 2FA for as many services as I could; some of it is particularly bothersome (LinkedIn, I’ll probably write more about that), but at least it’s an extra safety. Unfortunately, I still find it extremely bothersome neither Google Authenticator nor RedHat’s FreeOTP (last time I tried) supported backing up the private keys of the configured OTPs. Since I switched between three phones in the past three months, I could use some help when having to re-enroll my OTP generators — I upgraded my phone, then had to downgrade because I broke the screen of the new one.

Jeremy Olexa a.k.a. darkside (homepage, bugs)
Bypassing Geolocation … (September 07, 2014, 00:42 UTC)

By now we all know that it is pretty easy to bypass geolocation blockage with a web proxy or vpn service. After all, there is over 2 million google results on “bbc vpn” … and I wanted to do just that to view a BBC show on privacy and the dark web.

I wanted to set this up as cheaply as possible but not use a service that I had to pay for a month since I only needed one hour. This requirement directed me towards a do-it-yourself solution with an hourly server in the UK. I also wanted reproducibility so that I could spin up a similar service again in the future.

My first attempt was to route my browser through a local SOCKS proxy via ssh tunneling, ssh -D 2001 user@uk-host.tld. That didn’t work because my home connection was not good enough to stream from BBC without incessant buffering.

Hmm, if this simple proxy won’t work then that strikes out many other ideas, I needed a way to use the BBC iPlayer Downloader to view content offline. Ok, but the software doesn’t have native proxy support (naturally). Maybe you could somehow use TOR and set the exit node to the UK. That seems like a poor/slow idea.

I ended up routing all my traffic through a personal OpenVPN server in London and then downloaded the show via the BBC software and watched it in HD offline. The goal was to provision the VPN as quickly as possible (time is money). A Linode StackScript is a feature that Linode offers, it is a user defined script ran at first boot of your host. Surprisingly, no one published one to install OpenVPN yet. So, I did: “Debian 7.5 OpenVPN” – feel free to use it on the Linode service to boot up a vpn automatically. It takes about two minutes to boot, install, and configure OpenVPN this way. Then you download the ca.crt and client configuration from the newly provisioned server and import it into your client.

End result: It took 42 minutes for me to download a one hour show. Since I shut down the VPN within an hour, I was charged the Linode minimum, $.015 USD. Though I recommend Linode (you can use my referral link if you want), this same concept applies to any provider that has a presence in the UK, like Digital Ocean who charges $.007/hour.

Addendum: Even though I abandoned my first attempt, I left the browser window open and it continued to download even after I was disconnected from my UK VPN. I guess BBC only checks your IP once then hands you off to the Akamai CDN. Maybe you only need a VPN service for a few minutes?

I also donated money to a BBC sponsored charity to offset some of my bandwidth usage and freeloading of a service that UK citizens have to pay for, I encourage you to do that same. For reference it costs a UK household, $.02 USD tax per hour for BBC. (source)

September 06, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Anatomy of a security disaster (September 06, 2014, 22:28 UTC)

I have made a note of this in my previous post about Magnatune being terribly insecure. Those who follow me on Twitter or Google+ already got the full details of it but I thought I would repeat them here. And add a few notes about that.

I remember Magnatune back in the days in which I hang around #amarok and helped with small changes here and there, and bigger changes for xine itself. It was at the time often used as an example of good DRM-less services. Indeed, it sold DRM-free music way before Apple decided to drop its own music DRM, and its still one of the few services selling lossless music — if we exclude Humble Bundle and the games OSTs.

But then again, this is not a license to have terrible security, which is what Magnatune has right now. After naming Magnatune in my the aforementioned post I realized that I had not given it a new, good password but it’s instead still using one of the old passwords I used to use, which are both insecure by themselves, a bit too short, possibly suitable to dictionary attacks, and I was not even sure if it was using the password I used by default on many services before, which is of course terrible, and was most likely leaked at multiple points in time — at least my old Adobe account was involved in their big leak.

As I said before, I stopped using fixed passwords some time last year, and then I decided to jump on LastPass when Heartbleed required me to change passwords almost everywhere. But it takes a while to change passwords in all your accounts, especially when you forget about some accounts altogether, like the Magnatune one above.

So I went to Magnatune website to change my password, but of course I forgot what the original was, so I went on and decided to follow the procedure for forgotten passwords. The first problem happens here: it does not require me to know which email address I registered with, instead it asks me (alternatively) for an username, which is quite obvious (Flameeyes, what else? There are very few sites where I use different usernames, one of which being Tumblr, and that’s usually because Flameeyes is taken). When I type that in, it actually shows me on the web page the email address I’m registered with.

What? This is a basic privacy issue: if it wasn’t that I actually don’t vary my email addresses that much, an attacker could now find an otherwise private email address. Worse yet, by using the users available in previous dumps, it’s possible to assign them to email addresses, too. Indeed, A quick check provided me with at least one email address of a friend of mine by just using her usual username — I already knew the email address but that shouldn’t be a given.

Anyway, I got an email back from Magnatune just a moment later. The email contains the password in plain text, which indicates they store it that way, which is bad practice. A note about plain text passwords: there is no way to prove beyond any doubt that a service is hashing (or hashing and salting) user passwords, but you can definitely prove otherwise. If you receive your password back in plain text when you say you forgot it, then the service does not store hashed passwords. Conversely, even if the service sends you a password reset link instead, it’s still possible it’s storing the plain text password. This is most definitely unfortunate.

Up to here, things would be bad but not that uncommon, as the linked Plain Text Offenders site above would show you — and I have indeed submitted a screenshot of the email to them. But there is one more thing you can find out from the email they sent. You may remember that some months ago I wrote about email security and around the same time so did the Google Official blog – for those who wonder, no I had no idea that such a post was being written and the similar timing was a complete coincidence – so what’s the status of Magnatune? Well, unfortunately it’s bleak, as they don’t encrypt mail in transit:

Received: from magnatune.com ([64.62.194.219])
        by mx.google.com with ESMTP id h11si9367820pdl.64.2014.08.28.15.47.42
        for <f********@*****.***>;
        Thu, 28 Aug 2014 15:47:42 -0700 (PDT)

If the sending server spoke TLS to the GMail server (yes it’s gmail in the address I censored), it would have shown something like (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); (which appears in the comment messages I receive from my own blog).

Not encrypting the email in transit means that anybody that could have sniffed the traffic coming out of Magnatune’s server would be able to access any of their customers’ accounts: they just need to snoop the email traffic and they can receive all the password. Luckily, the email server from which the email arrived is hosted at a company I trust very much, as I’m their customer too.

So I tried logging in with my username and newly-reminded password, unfortunately my membership expired years ago, which means I get no access at all — so I can’t even change my password or my email address. Too bad. But then it allowed me to figure out some more problems with the security situation of Magnatune.

When you try to login, you get sent on a different website depending on which kind of membership you subscribe(d) to. In my case I got the download membership — when you go there, you get presented with a dialog requesting user and password from your browser. It’s standard HTTP based authentication. It’s not very common because it’s not really user friendly: you can’t serve any content until the user either puts the right username/password or decides they don’t know a valid combination and cancel the dialog, in which case a final 401 error is reported, and whichever content the server sent will be displayed by the browser.

Beside the userfriendliness (or lack thereof), HTTP authentication can be tricky, too. There are two ways to provide authentication over HTTP, one is Basic and the other is Digest — neither is very secure by default. Digest is partially usable, but suffer from lack of authentication of parties, making MitM attacks trivial, while Basic, well, allows a sniffer to figure out username and password as they travel in plaintext over the wire. HTTP authentication is, though, fairly secure if you use it in conjunction with TLS. Indeed for some of my systems I use HTTP authentication on a HTTPS connection, as it allows me to configure the authentication at the web server level without support from the application itself.

What became obvious to me while failing to log in to Magnatune was that the connection was not secure: it was cleartext HTTP that it was trying to get me to log in through. So I checked the headers to figure out which kind of authentication it was doing. At this point I have to use “of course” to say that it is using Basic authentication: cleartext username and password on the wire. This is extremely troublesome.

While not encrypting email reduces the attack surface, making it mostly a matter of people sniffing at the datacenter where Magnatune is hosted – assuming you use an email provider that is safe or trustworthy enough, I consider mine so – using basic authentication extend the surface immensely. Indeed, if you’re logging in Magnatune from a coffee shop or any other public open WiFi, you are literally broadcasting over the network your username and password.

I can’t know if you can change your Magnatune password once you can log in, since I can’t log in. But I know that the alternative to the download membership is the streaming membership, which makes it very likely that a Magnatune user would be logging in while at a Starbucks, so that they can work on some blog post or on source code of their mobile app while listening to music. I would hope they used a different password for Magnatune than for their email address — since as I noted above, you can get to their email address just by knowing their username.

I don’t worry too much. My Magnatune password turned out to be different enough from most of my other passwords that even if I don’t change it and gets leaked it won’t compromise any other service. Even more so now that I’m actively gathering all my account and changing their passwords.

September 05, 2014


Figure 1.1: Iron Penguin

Fig. 1: Iron Penguin

Gentoo Linux is proud to announce the availability of a new LiveDVD to celebrate the continued collaboration between Gentoo users and developers, The LiveDVD features a superb list of packages, some of which are listed below.

A special thanks to the Gentoo Infrastructure Team and likewhoa. Their hard work behind the scenes provide the resources, services and technology necessary to support the Gentoo Linux project.

  • Packages included in this release: Linux Kernel 3.15.6, Xorg 1.16.0, KDE 4.13.3, Gnome 3.12.2, XFCE 4.10, Fluxbox 1.3.5, LXQT Desktop 0.7.0, i3 Desktop 2.8, Firefox 31.0, LibreOffice 4.2.5.2, Gimp 2.8.10-r1, Blender 2.71-r1, Amarok 2.8.0-r2, Chromium 37.0.2062.35 and much more ...
  • If you want to see if your package is included we have generated both the x86 package list, and amd64 package list. The FAQ is located at FAQ. DVD cases and covers for the 20140826 release are located at Artwork. Persistence mode is back in the 20140826 release!.

The LiveDVD is available in two flavors: a hybrid x86/x86_64 version, and an x86_64 multi lib version. The livedvd-x86-amd64-32ul-20140826 version will work on 32-bit x86 or 64-bit x86_64. If your CPU architecture is x86, then boot with the default gentoo kernel. If your arch is amd64, boot with the gentoo64 kernel. This means you can boot a 64-bit kernel and install a customized 64-bit user land while using the provided 32-bit user land. The livedvd-amd64-multilib-20140826 version is for x86_64 only.

If you are ready to check it out, let our bouncer direct you to the closest x86 image or amd64 image file.

If you need support or have any questions, please visit the discussion thread on our forum.

Thank you for your continued support,
Gentoo Linux Developers, the Gentoo Foundation, and the Gentoo-Ten Project.

Michał Górny a.k.a. mgorny (homepage, bugs)
Bash pitfalls: globbing everywhere! (September 05, 2014, 08:31 UTC)

Bash has many subtle pitfalls, some of them being able to live unnoticed for a very long time. A common example of that kind of pitfall is ubiquitous filename expansion, or globbing. What many script writers forget about to notice is that practically anything that looks like a pattern and is not quoted is subject to globbing, including unquoted variables.

There are two extra snags that add up to this. Firstly, many people forget that not only asterisks (*) and question marks (?) make up patterns — square brackets ([) do that as well. Secondly, by default bash (and POSIX shell) take failed expansions literally. That is, if your glob does not match any file, you may not even know that you are globbing.

It's all just a matter of running in the proper directory for the result to change. Of course, it's often unlikely — maybe even close to impossible. You can work towards preventing that by running in a safe directory. But in the end, writing predictable software is a fine quality.

How to notice mistakes?

Bash provides a two major facilities that could help you stop mistakes — shopts nullglob and failglob.

The nullglob option is a good choice for a default for your script. After enabling it, failing filename expansions result in no parameters rather than verbatim pattern itself. This has two important implications.

Firstly, it makes iterating over optional files easy:

for f in a/* b/* c/*; do
    some_magic "${f}"
done

Without nullglob, the above may actually return a/* if no file matches the pattern. For this reason, you would need to add an additional check for existence of file inside the loop. With nullglob, it will just ‘omit’ the unmatched arguments. In fact, if none of the patterns match the loop won't be run even once.

Secondly, it turns every accidental glob into null. While this isn't the most friendly warning and in fact it may have very undesired results, you're more likely to notice that something is going wrong.

The failglob option is better if you can assume you don't need to match files in its scope. In this case, bash treats every failing filename expansion as a fatal error and terminates execution with an appropriate message.

The main advantage of failglob is that it makes you aware of any mistake before someone hits it the hard way. Of course, assuming that it doesn't accidentally expand into something already.

There is also a choice of noglob. However, I wouldn't recommend it since it works around mistakes rather than fixing them, and makes the code rely on a non-standard environment.

Word splitting without globbing

One of the pitfalls I myself noticed lately is the attempt of using unquoted variable substitution to do word splitting. For example:

for i in ${v}; do
    echo "${i}"
done

At a first glance, everything looks fine. ${v} contains a whitespace-separated list of words and we iterate over each word. The pitfall here is that words in ${v} are subject to filename expansion. For example, if a lone asterisk would happen to be there (like v='10 * 4'), you'd actually get all files in the current directory. Unexpected, isn't it?

I am aware of three solutions that can be used to accomplish word splitting without implicit globbing:

  1. setting shopt -s noglob locally,
  2. setting GLOBIGNORE='*' locally,
  3. using the swiss army knife of read to perform word splitting.

Personally, I dislike the first two since they require set-and-restore magic, and the latter also has the penalty of doing the globbing then discarding the result. Therefore, I will expand on using read:

read -r -d '' -a words <<<"${v}"
for i in "${words[@]}"; do
    echo "${i}"
done

While normally read is used to read from files, we can use the here string syntax of bash to feed the variable into it. The -r option disables backslash escape processing that is undesired here. -d '' causes read to process the whole input and not stop at any delimiter (like newline). -a words causes it to put the split words into array ${words[@]} — and since we know how to safely iterate over an array, the underlying issue is solved.

Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
32bit Madness (September 05, 2014, 06:41 UTC)

This week I ran into a funny issue doing backups with rsync:

rsnapshot/weekly.3/server/storage/lost/of/subdirectories/some-stupid.file => rsnapshot/daily.0/server/storage/lost/of/subdirectories/some-stupid.file
ERROR: out of memory in make_file [generator]
rsync error: error allocating core memory buffers (code 22) at util.c(117) [generator=3.0.9]
rsync error: received SIGUSR1 (code 19) at main.c(1298) [receiver=3.0.9]
rsync: connection unexpectedly closed (2168136360 bytes received so far) [sender]
rsync error: error allocating core memory buffers (code 22) at io.c(605) [sender=3.0.9]
Oopsiedaisy, rsync ran out of memory. But ... this machine has 8GB RAM, plus 32GB Swap ?!
So I re-ran this and started observing, and BAM, it fails again. With ~4GB RAM free.

4GB you say, eh? That smells of ... 2^32 ...
For doing the copying I was using sysrescuecd, and then it became obvious to me: All binaries are of course 32bit!

So now I'm doing a horrible hack of "linux64 chroot /mnt/server" so that I have a 64bit environment that does not run out of space randomly. Plus 3 new bugs for the Gentoo livecd, which fails to appreciate USB and other things.
Who would have thought that a 16TB partition can make rsync stumble over address space limits ...

September 03, 2014
Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
AMD HSA (September 03, 2014, 06:25 UTC)

With the release of the "Kaveri" APUs AMD has released some quite intriguing technology. The idea of the "APU" is a blend of CPU and GPU, what AMD calls "HSA" - Heterogenous System Architecture.
What does this mean for us? In theory, once software catches up, it'll be a lot easier to use GPU-acceleration (e.g. OpenCL) within normal applications.

One big advantage seems to be that CPU and GPU share the system memory, so with the right drivers you should be able to do zero-copy GPU processing. No more host-to-GPU copy and other waste of time.

So far there hasn't been any driver support to take advantage of that. Here's the good news: As of a week or two ago there is driver support. Still very alpha, but ... at last, drivers!

On the kernel side there's the kfd driver, which piggybacks on radeon. It's available in a slightly very patched kernel from AMD. During bootup it looks like this:

[    1.651992] [drm] radeon kernel modesetting enabled.
[    1.657248] kfd kfd: Initialized module
[    1.657254] Found CRAT image with size=1440
[    1.657257] Parsing CRAT table with 1 nodes
[    1.657258] Found CU entry in CRAT table with proximity_domain=0 caps=0
[    1.657260] CU CPU: cores=4 id_base=16
[    1.657261] Found CU entry in CRAT table with proximity_domain=0 caps=0
[    1.657262] CU GPU: simds=32 id_base=-2147483648
[    1.657263] Found memory entry in CRAT table with proximity_domain=0
[    1.657264] Found memory entry in CRAT table with proximity_domain=0
[    1.657265] Found memory entry in CRAT table with proximity_domain=0
[    1.657266] Found memory entry in CRAT table with proximity_domain=0
[    1.657267] Found cache entry in CRAT table with processor_id=16
[    1.657268] Found cache entry in CRAT table with processor_id=16
[    1.657269] Found cache entry in CRAT table with processor_id=16
[    1.657270] Found cache entry in CRAT table with processor_id=17
[    1.657271] Found cache entry in CRAT table with processor_id=18
[    1.657272] Found cache entry in CRAT table with processor_id=18
[    1.657273] Found cache entry in CRAT table with processor_id=18
[    1.657274] Found cache entry in CRAT table with processor_id=19
[    1.657274] Found TLB entry in CRAT table (not processing)
[    1.657275] Found TLB entry in CRAT table (not processing)
[    1.657276] Found TLB entry in CRAT table (not processing)
[    1.657276] Found TLB entry in CRAT table (not processing)
[    1.657277] Found TLB entry in CRAT table (not processing)
[    1.657278] Found TLB entry in CRAT table (not processing)
[    1.657278] Found TLB entry in CRAT table (not processing)
[    1.657279] Found TLB entry in CRAT table (not processing)
[    1.657279] Found TLB entry in CRAT table (not processing)
[    1.657280] Found TLB entry in CRAT table (not processing)
[    1.657286] Creating topology SYSFS entries
[    1.657316] Finished initializing topology ret=0
[    1.663173] [drm] initializing kernel modesetting (KAVERI 0x1002:0x1313 0x1002:0x0123).
[    1.663204] [drm] register mmio base: 0xFEB00000
[    1.663206] [drm] register mmio size: 262144
[    1.663210] [drm] doorbell mmio base: 0xD0000000
[    1.663211] [drm] doorbell mmio size: 8388608
[    1.663280] ATOM BIOS: 113
[    1.663357] radeon 0000:00:01.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF (1024M used)
[    1.663359] radeon 0000:00:01.0: GTT: 1024M 0x0000000040000000 - 0x000000007FFFFFFF
[    1.663360] [drm] Detected VRAM RAM=1024M, BAR=256M
[    1.663361] [drm] RAM width 128bits DDR
[    1.663471] [TTM] Zone  kernel: Available graphics memory: 7671900 kiB
[    1.663472] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[    1.663473] [TTM] Initializing pool allocator
[    1.663477] [TTM] Initializing DMA pool allocator
[    1.663496] [drm] radeon: 1024M of VRAM memory ready
[    1.663497] [drm] radeon: 1024M of GTT memory ready.
[    1.663516] [drm] Loading KAVERI Microcode
[    1.667303] [drm] Internal thermal controller without fan control
[    1.668401] [drm] radeon: dpm initialized
[    1.669403] [drm] GART: num cpu pages 262144, num gpu pages 262144
[    1.685757] [drm] PCIE GART of 1024M enabled (table at 0x0000000000277000).
[    1.685894] radeon 0000:00:01.0: WB enabled
[    1.685905] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr 0x0000000040000c00 and cpu addr 0xffff880429c5bc00
[    1.685908] radeon 0000:00:01.0: fence driver on ring 1 use gpu addr 0x0000000040000c04 and cpu addr 0xffff880429c5bc04
[    1.685910] radeon 0000:00:01.0: fence driver on ring 2 use gpu addr 0x0000000040000c08 and cpu addr 0xffff880429c5bc08
[    1.685912] radeon 0000:00:01.0: fence driver on ring 3 use gpu addr 0x0000000040000c0c and cpu addr 0xffff880429c5bc0c
[    1.685914] radeon 0000:00:01.0: fence driver on ring 4 use gpu addr 0x0000000040000c10 and cpu addr 0xffff880429c5bc10
[    1.686373] radeon 0000:00:01.0: fence driver on ring 5 use gpu addr 0x0000000000076c98 and cpu addr 0xffffc90012236c98
[    1.686375] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    1.686376] [drm] Driver supports precise vblank timestamp query.
[    1.686406] radeon 0000:00:01.0: irq 83 for MSI/MSI-X
[    1.686418] radeon 0000:00:01.0: radeon: using MSI.
[    1.686441] [drm] radeon: irq initialized.
[    1.689611] [drm] ring test on 0 succeeded in 3 usecs
[    1.689699] [drm] ring test on 1 succeeded in 2 usecs
[    1.689712] [drm] ring test on 2 succeeded in 2 usecs
[    1.689849] [drm] ring test on 3 succeeded in 2 usecs
[    1.689856] [drm] ring test on 4 succeeded in 2 usecs
[    1.711523] tsc: Refined TSC clocksource calibration: 3393.828 MHz
[    1.746010] [drm] ring test on 5 succeeded in 1 usecs
[    1.766115] [drm] UVD initialized successfully.
[    1.767829] [drm] ib test on ring 0 succeeded in 0 usecs
[    2.268252] [drm] ib test on ring 1 succeeded in 0 usecs
[    2.712891] Switched to clocksource tsc
[    2.768698] [drm] ib test on ring 2 succeeded in 0 usecs
[    2.768819] [drm] ib test on ring 3 succeeded in 0 usecs
[    2.768870] [drm] ib test on ring 4 succeeded in 0 usecs
[    2.791599] [drm] ib test on ring 5 succeeded
[    2.812675] [drm] Radeon Display Connectors
[    2.812677] [drm] Connector 0:
[    2.812679] [drm]   DVI-D-1
[    2.812680] [drm]   HPD3
[    2.812682] [drm]   DDC: 0x6550 0x6550 0x6554 0x6554 0x6558 0x6558 0x655c 0x655c
[    2.812683] [drm]   Encoders:
[    2.812684] [drm]     DFP2: INTERNAL_UNIPHY2
[    2.812685] [drm] Connector 1:
[    2.812686] [drm]   HDMI-A-1
[    2.812687] [drm]   HPD1
[    2.812688] [drm]   DDC: 0x6530 0x6530 0x6534 0x6534 0x6538 0x6538 0x653c 0x653c
[    2.812689] [drm]   Encoders:
[    2.812690] [drm]     DFP1: INTERNAL_UNIPHY
[    2.812691] [drm] Connector 2:
[    2.812692] [drm]   VGA-1
[    2.812693] [drm]   HPD2
[    2.812695] [drm]   DDC: 0x6540 0x6540 0x6544 0x6544 0x6548 0x6548 0x654c 0x654c
[    2.812695] [drm]   Encoders:
[    2.812696] [drm]     CRT1: INTERNAL_UNIPHY3
[    2.812697] [drm]     CRT1: NUTMEG
[    2.924144] [drm] fb mappable at 0xC1488000
[    2.924147] [drm] vram apper at 0xC0000000
[    2.924149] [drm] size 9216000
[    2.924150] [drm] fb depth is 24
[    2.924151] [drm]    pitch is 7680
[    2.924428] fbcon: radeondrmfb (fb0) is primary device
[    2.994293] Console: switching to colour frame buffer device 240x75
[    2.999979] radeon 0000:00:01.0: fb0: radeondrmfb frame buffer device
[    2.999981] radeon 0000:00:01.0: registered panic notifier
[    3.008270] ACPI Error: [\_SB_.ALIB] Namespace lookup failure, AE_NOT_FOUND (20131218/psargs-359)
[    3.008275] ACPI Error: Method parse/execution failed [\_SB_.PCI0.VGA_.ATC0] (Node ffff88042f04f028), AE_NOT_FOUND (20131218/psparse-536)
[    3.008282] ACPI Error: Method parse/execution failed [\_SB_.PCI0.VGA_.ATCS] (Node ffff88042f04f000), AE_NOT_FOUND (20131218/psparse-536)
[    3.509149] kfd: kernel_queue sync_with_hw timeout expired 500
[    3.509151] kfd: wptr: 8 rptr: 0
[    3.509243] kfd kfd: added device (1002:1313)
[    3.509248] [drm] Initialized radeon 2.37.0 20080528 for 0000:00:01.0 on minor 0
It is recommended to add udev rules:
# cat /etc/udev/rules.d/kfd.rules 
KERNEL=="kfd", MODE="0666"
(this might not be the best way to do it, but we're just here to test if things work at all ...)

AMD has provided a small shell script to test if things work:
# ./kfd_check_installation.sh 

Kaveri detected:............................Yes
Kaveri type supported:......................Yes
Radeon module is loaded:....................Yes
KFD module is loaded:.......................Yes
AMD IOMMU V2 module is loaded:..............Yes
KFD device exists:..........................Yes
KFD device has correct permissions:.........Yes
Valid GPU ID is detected:...................Yes

Can run HSA.................................YES
So that's a good start. Then you need some support libs ... which I've ebuildized in the most horrible ways
These ebuilds can be found here

Since there's at least one binary file with undeclared license and some other inconsistencies I cannot recommend installing these packages right now.
And of course I hope that AMD will release the sourcecode of these libraries ...

There's an example "vector_copy" program included, it mostly works, but appears to go into an infinite loop. Outout looks like this:
# ./vector_copy 
Initializing the hsa runtime succeeded.
Calling hsa_iterate_agents succeeded.
Checking if the GPU device is non-zero succeeded.
Querying the device name succeeded.
The device name is Spectre.
Querying the device maximum queue size succeeded.
The maximum queue size is 131072.
Creating the queue succeeded.
Creating the brig module from vector_copy.brig succeeded.
Creating the hsa program succeeded.
Adding the brig module to the program succeeded.
Finding the symbol offset for the kernel succeeded.
Finalizing the program succeeded.
Querying the kernel descriptor address succeeded.
Creating a HSA signal succeeded.
Registering argument memory for input parameter succeeded.
Registering argument memory for output parameter succeeded.
Finding a kernarg memory region succeeded.
Allocating kernel argument memory buffer succeeded.
Registering the argument buffer succeeded.
Dispatching the kernel succeeded.
^C
Big thanks to AMD for giving us geeks some new toys to work with, and I hope it becomes a reliable and efficient platform to do some epic numbercrunching :)

August 30, 2014
Sven Vermeulen a.k.a. swift (homepage, bugs)
Showing return code in PS1 (August 30, 2014, 23:14 UTC)

If you do daily management on Unix/Linux systems, then checking the return code of a command is something you’ll do often. If you do SELinux development, you might not even notice that a command has failed without checking its return code, as policies might prevent the application from showing any output.

To make sure I don’t miss out on application failures, I wanted to add the return code of the last executed command to my PS1 (i.e. the prompt displayed on my terminal).
I wasn’t able to add it to the prompt easily – in fact, I had to use a bash feature called the prompt command.

When the PROMPT_COMMMAND variable is defined, then bash will execute its content (which I declare as a function) to generate the prompt. Inside the function, I obtain the return code of the last command ($?) and then add it to the PS1 variable. This results in the following code snippet inside my ~/.bashrc:

export PROMPT_COMMAND=__gen_ps1
 
function __gen_ps1() {
  local EXITCODE="$?";
  # Enable colors for ls, etc.  Prefer ~/.dir_colors #64489
  if type -P dircolors >/dev/null ; then
    if [[ -f ~/.dir_colors ]] ; then
      eval $(dircolors -b ~/.dir_colors)
    elif [[ -f /etc/DIR_COLORS ]] ; then
      eval $(dircolors -b /etc/DIR_COLORS)
    fi
  fi
 
  if [[ ${EUID} == 0 ]] ; then
    PS1="RC=${EXITCODE} \[\033[01;31m\]\h\[\033[01;34m\] \W \$\[\033[00m\] "
  else
    PS1="RC=${EXITCODE} \[\033[01;32m\]\u@\h\[\033[01;34m\] \w \$\[\033[00m\] "
  fi
}

With it, my prompt now nicely shows the return code of the last executed command. Neat.

Edit: Sean Patrick Santos showed me my utter failure in that this can be accomplished with the PS1 variable immediately, without using the overhead of the PROMPT_COMMAND. Just make sure to properly escape the $ sign which I of course forgot in my late-night experiments :-(.

Luca Barbato a.k.a. lu_zero (homepage, bugs)
PowerPC is back (and little endian) (August 30, 2014, 17:32 UTC)

Yesterday I fixed a PowerPC issue since ages, it is an endianess issue, and it is (funny enough) on the little endian flavour of it.

PowerPC

I have some ties with this architecture since my interest on the architecture (and Altivec/VMX in particular) is what made me start contributing to MPlayer while fixing issue on Gentoo and from there hack on the FFmpeg of the time, meet the VLC people, decide to part ways with Michael Niedermayer and with the other main contributors of FFmpeg create Libav. Quite a loong way back in the time.

Big endian, Little Endian

It is a bit surprising that IBM decided to use little endian (since big endian is MUCH nicer for I/O processing such as networking) but they might have their reasons.

PowerPC traditionally always had been both-endian with the ability to switch on the fly between the two (this made having foreign-endian simulators lightly less annoying to manage), but the main endianess had always been big.

This brings us to a quite interesting problem: Some if not most of the PowerPC code had been written thinking in big-endian. Luckily since most of the code wrote was using C intrinsics (Bless to whoever made the Altivec intrinsics not as terrible as the other ones around) it won’t be that hard to recycle most of the code.

More will follow.

August 29, 2014
Sven Vermeulen a.k.a. swift (homepage, bugs)
Gentoo Hardened august meeting (August 29, 2014, 14:43 UTC)

Another month has passed, so we had another online meeting to discuss the progress within Gentoo Hardened.

Lead elections

The yearly lead elections within Gentoo Hardened were up again. Zorry (Magnus Granberg) was re-elected as project lead so doesn’t need to update his LinkedIn profile yet ;-)

Toolchain

blueness (Anthony G. Basile) has been working on the uclibc stages for some time. Due to the configurable nature of these setups, many /etc/portage files were provided as part of the stages, which shouldn’t happen. Work is on the way to update this accordingly.

For the musl setup, blueness is also rebuilding the stages to use a symbolic link to the dynamic linker (/lib/ld-linux-arch.so) as recommended by the musl maintainers.

Kernel and grsecurity with PaX

A bug has been submitted which shows that large binary files (in the bug, a chrome binary with debug information is shown to be more than 2 Gb in size) cannot be pax-mark’ed, with paxctl informing the user that the file is too big. The problem is when the PAX marks are in ELF (as the application mmaps the binary) – users of extended attributes based PaX markings do not have this problem. blueness is working on making things a bit more intelligent, and to fix this.

SELinux

I have been making a few changes to the SELinux setup:

  • The live ebuilds (those with version 9999 which use the repository policy rather than snapshots of the policies) are now being used as “master” in case of releases: the ebuilds can just be copied to the right version to support the releases. The release script inside the repository is adjusted to reflect this as well.
  • The SELinux eclass now supports two variables, SELINUX_GIT_REPO and SELINUX_GIT_BRANCH, which allows users to use their own repository, and developers to work in specific branches together. By setting the right value in the users’ make.conf switching policy repositories or branches is now a breeze.
  • Another change in the SELinux eclass is that, after the installation of SELinux policies, we will check the reverse dependencies of the policy package and relabel the files of these packages. This allows us to only have RDEPEND dependencies towards the SELinux policy packages (if the application itself does not otherwise link with libselinux), making the dependency tree within the package manager more correct. We still need to update these packages to drop the DEPEND dependency, which is something we will focus on in the next few months.
  • In order to support improved cooperation between SELinux developers in the Gentoo Hardened team – perfinion (Jason Zaman) is in the queue for becoming a new developer in our mids – a coding style for SELinux policies is being drafted up. This is of course based on the coding style of the reference policy, but with some Gentoo specific improvements and more clarifications.
  • perfinion has been working on improving the SELinux support in OpenRC (release 0.13 and higher), making some of the additions that we had to make in the past – such as the selinux_gentoo init script – obsolete.

The meeting also discussed a few bugs in more detail, but if you really want to know, just hang on and wait for the IRC logs ;-) Other usual sections (system integrity and profiles) did not have any notable topics to describe.

August 28, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
Did Apple lose its advantage? (August 28, 2014, 22:59 UTC)

Readers of my blog for a while probably know already that I've been an Apple user over time. What is not obvious is that I have scaled down my (personal) Apple usage over the past two years, mostly because my habits, and partly because of Android and Linux getting better and better. One component is, though, that some of the advantages to be found when using Apple started to disappear for me.

I think that for me the start of the problems is to be found in the release of iOS 7. Beside the taste of not liking the new flashy UI, what I found is that it did not perform as well as previous releases. I think this is the same effect others have had. In particular the biggest problem with it for me had to do with the way I started using my iPad while in Ireland. Since I now have access to a high-speed connection, I started watching more content in streaming. In particular, thanks to my multiple trips to the USA over the past year, I got access to more video content on the iTunes store, so I wanted to watch some of the new TV series through it.

Turned out that for a few versions, and I mean a few months, iOS was keeping the streamed content in the cache, not accounting for it anywhere, and never cleaning it up. The result was that after streaming half a series, I would get errors telling me the iPad storage was full, but there was no way from the device itself to clear the cache. EIther you had to do a factory reset to drop off all the content of the device, or you had to use a Windows application to remove the cache files manually. Not very nice.

Another very interesting problem with the streaming the content: it can be slow. Not always but it can. One night I wanted to watch The LEGO Movie since I did not see it at the cinema. It's not available on the Irish Netflix so I decided to rent it off iTunes. It took the iPad four hours to download it. It made no sense. And no, the connection was not hogged by something else, and running a SpeedTest from the tablet itself showed it had all the network capacity it needed.

The iPad is not, though, the only Apple device I own; I also bought an iPod Touch back in LA when my Classic died. even though I was not really happy with downgrading from 80G down to 64G. But it's mostly okay, as my main use for the iPod is to listen to audiobooks and podcasts when I sleep — which recently I have been doing through Creative D80 Bluetooth speakers, which are honestly not great but at least don't force me to wear earphones all night long.

I had no problem before switching the iPod from one computer to the next, as I moved from iMac to a Windows disk for my laptop. When I decided to just use iTunes on the one Windows desktop I keep around (mostly to play games), then a few things stopped working as intended. It might have been related to me dropping the iTunes Match subscription, but I'm not sure about that. But what happens is that only a single track for each of the albums was being copied on the iPod and nothing else.

I tried factory reset, cable and wireless sync, I tried deleting the iTunes data on my computer to force it to figure out the iPod is new, and the current situation I'm in is only partially working: the audiobooks have been synced, but without cover art and without the playlists — some of the audiobooks I have are part of a series, or are split in multiple files if I bought them before Audible started providing single-file downloads. This is of course not very good when the audio only lasts three hours, and then I start having nightmares.

It does not help that I can't listen to my audiobooks with VLC for Android because it thinks that the chapter art is a video stream, and thus puts the stream to pause as soon as I turn off the screen. I should probably write a separate rant about the lack of proper audiobooks tools for Android. Audible has an app, but it does not allow you to sideload audiobooks (i.e. stuff I ripped from my original CDs, or that I bought on iTunes), nor it allows you to build a playlist of books, say for all the books in a series.

As I write this, I asked iTunes again to sync all the music to my iPod Touch as 128kbps AAC files (as otherwise it does not fit into the device); iTunes is now copying 624 files; I'm sure my collection contains more than 600 albums — and I would venture to say more than half I have in physical media. Mostly because no store allows me to buy metal in FLAC or ALAC. And before somebody suggests Jamendo or other similar services: yes, great, I actually bought lots of Jazz on Magnatune before it became a subscription service and I loved it, but that is not a replacement for mainstream content. Also, Magnatune has terrible security practices, don't use it.

Sorry Apple, but given these small-but-not-so-small issues with your software recently, I'm not going to buy any more devices from you. If any of the two devices I have fails, I'll just get someone to build a decent audiobook software for me one way or the other…

August 25, 2014
Matthew Thode a.k.a. prometheanfire (homepage, bugs)
Gentoo on the Odroid-U3 (August 25, 2014, 05:00 UTC)

Arm cross compiler setup and stuffs

This will set up a way to compile things for arm on your native system (amd64 for me)

emerge dev-embedded/u-boot-tools sys-devel/crossdev
crossdev -S -s4 -t armv7a-hardfloat-linux-gnueabi

Building the kernel

This assumes you have kernel sources, I'm testing 3.17-rc2 since they just got support for the odroid-u3 into upstream.

Also, I tend to build without modules, so keep that in mind.

# get the base config (For me on an odroid-u3
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make exynos_defconfig
# change it to add what I want/need
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make menuconfig
# build the kernel
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make -j10

Setting up the SD Card

I tend to be generous, 10M for the bootloader

parted /dev/sdb mklabel msdos y
parted /dev/sdb mkpart p fat32 10M 200M
parted /dev/sdb mkpart p 200M 100%
parted /dev/sdb toggle 1 boot

mkfs.vfat /dev/sdb1
mkfs.ext4 /dev/sdb2

Building uboot

This may differ between boards, but should general look like the following (I hear vanilla uboot works now)

I used the odroid-v2010.12 branch and one thing to note is that if it sees a zImage on the boot partition it will ONLY use that, kinda of annoying.

git clone git://github.com/hardkernel/u-boot.git
cd u-boot
sed -i -e "s/soft-float/float-abi=hard -mfpu=vfpv3/g" arch/arm/cpu/armv7/config.mk
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make smdk4412_config
ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- make -j1
sudo "sh /home/USER/dev/arm/u-boot/sd_fuse/sd_fusing.sh /dev/sdb"

Copying the kernel/userland

sudo -i
mount /dev/sdb2 /mnt/gentoo
mount /dev/sdb1 /mnt/gentoo/boot
cp /home/USER/dev/linux/arch/arm/boot/dts/exynos4412-odroidu3.dtb /mnt/gentoo/boot/
cp /home/USER/dev/linux/arch/arm/boot/zImage /mnt/gentoo/boot/kernel-3.17-rc2.raw
cd /mnt/gentoo/boot
cat kernel-3.17-rc2.raw exynos4412-odroidu3.dtb > kernel-3.17-rc2

tar -xf /tmp/stage3-armv7a_hardfp-hardened-20140627.tar.bz2 -C /mnt/gentoo/

Setting up userland

I tend to just copy or generate a shadow file and overwrite the root entry in /etc/shadow...

Then set up on when booted

Setting up the bootloader

put this in /mnt/gentoo/boot/boot.txt

setenv initrd_high "0xffffffff"
setenv fdt_high "0xffffffff"
setenv fb_x_res "1920"
setenv fb_y_res "1080"
setenv hdmi_phy_res "1080"
setenv bootcmd "fatload mmc 0:1 0x40008000 kernel-3.17-rc2; bootm 0x40008000"
setenv bootargs "console=tty1 console=ttySAC1,115200n8 fb_x_res=${fb_x_res} fb_y_res=${fb_y_res} hdmi_phy_res=${hdmi_phy_res} root=/dev/mmcblk0p2 rootwait ro mem=2047M"
boot

and run this

mkimage -A arm -T script -C none -n "Boot.scr for odroid-u3" -d boot.txt boot.scr

That should do it :D

I used steev (a fellow gentoo dev) and http://www.funtoo.org/ODROID_U2 as sources.

August 22, 2014
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

As of today, more than 50% of the 37527 ebuilds in the Gentoo portage tree use the newest ebuild API (EAPI) version, EAPI=5!
The details of the various EAPIs can be found in the package manager specification (PMS); the most notable new feature of EAPI 5, which has sped up acceptance a lot is the introduction of so-called subslots. A package A can specify a subslot, another package B that depends on it can specify that it needs to be rebuilt when the subslot of A changes. This leads to much more elegant solutions for many of the the link or installation path problems that revdep-rebuild, emerge @preserved-rebuild, or e.g. perl-cleaner try to solve... Another useful new feature in EAPI=5 is the masking of use-flags specifically for stable-marked ebuilds.
You can follow the adoption of EAPIs in the portage tree on an automatically updated graph page.

August 19, 2014
Sven Vermeulen a.k.a. swift (homepage, bugs)
Switching to new laptop (August 19, 2014, 20:11 UTC)

I’m slowly but surely starting to switch to a new laptop. The old one hasn’t completely died (yet) but given that I had to force its CPU frequency at the lowest Hz or the CPU would burn (and the system suddenly shut down due to heat issues), and that the connection between the battery and laptop fails (so even new battery didn’t help out) so I couldn’t use it as a laptop… well, let’s say the new laptop is welcome ;-)

Building Gentoo isn’t an issue (having only a few hours per day to work on it is) and while I’m at it, I’m also experimenting with EFI (currently still without secure boot, but with EFI) and such. Considering that the Gentoo Handbook needs quite a few updates (and I’m thinking to do more than just small updates) knowing how EFI works is a Good Thing ™.

For those interested – the EFI stub kernel instructions in the article on the wiki, and also in Greg’s wonderful post on booting a self-signed Linux kernel (which I will do later) work pretty well. I didn’t try out the “Adding more kernels” section in it, as I need to be able to (sometimes) edit the boot options (which isn’t easy to accomplish with EFI stub-supporting kernels afaics). So I installed Gummiboot (and created a wiki article on it).

Lots of things still planned, so little time. But at least building chromium is now a bit faster – instead of 5 hours and 16 minutes, I can now enjoy the newer versions after little less than 40 minutes.

August 17, 2014
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
What's up with Semalt, then? (August 17, 2014, 18:05 UTC)

In my previous post on the matter, I called for a boycott of Semalt by blocking access to your servers from their crawler, after a very bad-looking exchange on Twitter with a supposed representative of theirs.

After I posted that, I got threatened by the same representative to be sued for libel, even though what that post was about was documenting their current practices, rather than shaming them. This got enough attention of other people who has been following the Semalt situation so that I could actually gather some more information on the matter.

In particular, there are two interesting blog posts by Joram van den Boezen about the company and its tactics. Turns out that what I thought was a very strange private cloud set up – coming as it was from Malaysia – was actually a botnet. Indeed, what appears from Joram's investigations is that the people behind Semalt use sidecar malware both to gather URLs to crawl, and to crawl them. And this, according to their hosting provider is allowed because they make it clear in their software's license.

This is consistent with what I have seen of Semalt on my server: rather than my blog – which fares pretty well on the web as a source of information – I found them requesting my website, which is almost dead. Looking at all the websites in all my servers, the only other affected is my friend's which is by far not really an important one. But if we start from accepting Joram's findings (and I have no reason not to), then I can see how that can happen.

My friend's website is visited mostly by the people in the area we grew up in, and general friends of his. I know how bad their computers can be, as I have been doing tech support on them for years, and paid my bills that way. Computers that were bought either without a Windows license or with Windows Vista, that got XP installed on them so badly that they couldn't get updates even when they were available. Windows 7 updates that were done without actually possessing a license, and so on so forth. I have, at some point, added a ModRewrite-based warning for a few known viruses that would alter the Internet Explorer User-Agent field.

Add to this that even those who shouldn't be strapped for cash would want to avoid paying for anything if they can, you can see why software such as SoundFrost and other similar "tools" to download YouTube videos into music files would be quite likely to be found in computers that end up browsing my friend's site.

What remains still not clear from all this information is why they are doing it. As I said in my previous post, there is no reason to abuse the referrer field, that is, beside to spam the statistics of the websites. Since the company is selling SEO services, one assumes that they do so to attract more customers. After all, if you spend time checking your Analytics output, you probably are the target audience of SEO services.

But after that, there are still questions that have no answer. How can that company do any analytics when they don't really seem to have any infrastructure but rather use botnets for finding and accessing websites? Do they only make money with their subscriptions? And here is where things can get tricky, because I can only hypothesize and speculate, words that are dangerous to begin with.

What I can tell you is that out there, many people have no scruple, and I'm not referring to Semalt here. When I tried to raise awareness about them on Reddit (a site that I don't generally like, but that can be put to good use sometimes), I stopped by the subreddit to get an idea of what kind of people would be around there. It was not as I was expecting, not at all. Indeed what I found is that there are people out there seriously considering using black hat SEO services. Again, this is speculation, but my assumption is that these are consultants that basically want to show their clients that their services are worth it by inflating the access statistics to the websites.

So either these consultants just buy the services out of companies like Semalt, or even the final site owners don't understand that a company promising "more accesses" does not really mean "more people actually looking at your website and considering your services". It's hard for people who don't understand the technology to discern between "accesses" and "eyeballs'. It's not much different from the fake Twitter followers, studied by Barracuda Labs a couple of years ago — I know I read a more thorough study of one of the websites selling this kind of money but I can't find it. That's why I usually keep that stuff on Readability.

So once again, give some antibiotics to the network, and help cure the web from people like Semalt and the people who would buy their services.

August 16, 2014
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Libav Release Process (August 16, 2014, 15:23 UTC)

Since the release document is lacking here few notes on how it works, it will be updated soon =).

Versioning

Libav has separate version for each library provided. As usual the major version bump signifies an ABI-incompatible change, a minor version bump marks a specific feature introduction or removal.
It is made this way to let users leverage the pkgconf checks to require features instead of use a compile+link check.
The APIChange document details which version corresponds to which feature.

The Libav global version number e.g. 9.16 provides mainly the following information:

  • If the major number is updated the Libraries have ABI differences.
    • If the major number is Even API-incompatible changes should be expected, downstreams should follow the migration guide to update their code.
    • If the major number is Odd no API-incompatible changes happened and a simple rebuild **must** be enough to use the new library.
  • If the minor number is updated that means that enough bugfixes piled up during the month/2weeks period and a new point release is available.

Major releases

All the major releases start with a major version bump of all the libraries. This automatically enables new ABI incompatible code and disables old deprecated code. Later or within the same patch the preprocessor guards and the deprecated code gets removed.

Alpha

Once the major bump is committed the first alpha is tagged. Alphas live within the master branch, the codebase can still accept features updates (e.g. small new decoders or new demuxers) but the API and ABI cannot have incompatible changes till the next one or two major releases.

Beta

The first beta tag also marks the start of the new release branch.
From this point all the bugfixes that hit the master will be backported, no feature changes are accepted in the branch.

Release

The release is not different from a beta, it is still a tag in the release branch. The level of confidence nothing breaks is much higher though.

Point releases

Point releases are bugfix-only releases and they aim to provide seamless security updates.

Since most bugs in Libav are security concerns users should update as soon the new release is out. We keep our continuous integration system monitoring all the release branches in addition to the master branch to be confident that backported bugfixes do not cause unexpected issues.

Libav 11

The first beta for the release 11 should appear in the next two days, please help us by testing and reporting bugs.

August 14, 2014
Alexys Jacob a.k.a. ultrabug (homepage, bugs)

Foreword

Let’s say we have to design an application that should span across multiple datacenters while being able to scale as easily as firing up a new vm/container without the need to update any kind of configuration.

Facing this kind of challenge is exciting and requires us to address a few key scaffolding points before actually starting to code something :

  • having a robust and yet versatile application container to run our application
  • having a datacenter aware, fault detecting and service discovery service

Seeing the title of this article, the two components I’ll demonstrate are obviously uWSGI and Consul which can now work together thanks to the uwsgi-consul plugin.

While this article example is written in python, you can benefit from the same features in all the languages supported by uWSGI which includes go, ruby, perl ad php !

Our first service discovering application

The application will demonstrate how simple it is for a client to discover all the available servers running a specific service on a given port. The best part is that the services will be registered and deregistered automatically by uWSGI as they’re loaded and unloaded.

The demo application logic is as follows :

  1. uWSGI will load two server applications which are each responsible for providing the specified service on the given port
  2. uWSGI will automatically register the configured service into Consul
  3. uWSGI will also automatically register a health check for the configured service into Consul so that Consul will also be able to detect any failure of the service
  4. Consul will then respond to any client requesting the list of the available servers (nodes) providing the specified service
  5. The client will query Consul for the service and get either an empty response (no server available / loaded) or the list of the available servers

Et voilà, the client can dynamically detect new/obsolete servers and start working !

Setting up uWSGI and its Consul plugin

On Gentoo Linux, you’ll just have to run the following commands to get started (other users refer to the uWSGI documentation or your distro’s package manager). The plugin will be built by hand as I’m still not sure how I’ll package the uWSGI external plugins…

$ sudo ACCEPT_KEYWORDS="~amd64" emerge uwsgi
$ cd /usr/lib/uwsgi/
$ sudo uwsgi --build-plugin https://github.com/unbit/uwsgi-consul
$ cd -

 

You’ll have installed the uwsgi-consul plugin which you should see here :

$ ls /usr/lib/uwsgi/consul_plugin.so
/usr/lib/uwsgi/consul_plugin.so

 

That’s all we need to have uWSGI working with Consul.

Setting up a Consul server

Gentoo users will need to add the ultrabug overlay (use layman) and then install consul (other users refer to the Consul documentation or your distro’s package manager).

$ sudo layman -a ultrabug
$ sudo ACCEPT_KEYWORDS="~amd64" USE="web" emerge consul

 

Running the server and its UI is also quite straightforward. For this example, we will run it directly from a dedicated terminal so you can also enjoy the logs and see what’s going on (Gentoo users have an init script and conf.d ready for them shall they wish to go further).

Open a new terminal and run :

$ consul agent -data-dir=/tmp/consul-agent -server -bootstrap -ui-dir=/var/lib/consul/ui -client=0.0.0.0

 

You’ll see consul running and waiting for work. You can already enjoy the web UI by pointing your browser to http://127.0.0.1:8500/ui/.

Running the application

To get this example running, we’ll use the uwsgi-consul-demo code that I prepared.

First of all we’ll need the consulate python library (available on pypi via pip). Gentoo users can just install it (also from the ultrabug overlay added before) :

$ sudo ACCEPT_KEYWORDS="~amd64" emerge consulate

 

Now let’s clone the demo repository and get into the project’s directory.

$ git clone git@github.com:ultrabug/uwsgi-consul-demo.git
$ cd uwsgi-consul-demo

 

First, we’ll run the client which should report that no server is available yet. We will keep this terminal open to see the client detecting in real time the appearance and disappearance of the servers as we start and stop uwsgi :

$ python client.py 
no consul-demo-server available
[...]
no consul-demo-server available

 

Open a new terminal and get inside the project’s directory. Let’s have uWSGI load the two servers and register them in Consul :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2
[...]
* server #1 is up on port 2001


* server #2 is up on port 2002

[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully
[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully

 

Now let’s check back our client terminal, hooray it has discovered the two servers on the host named drakar (that’s my local box) !

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001

Expanding our application

Ok it works great on our local machine but we want to see how to add more servers to the fun and scale dynamically.

Let’s add another machine (named cheetah here) to the fun and have servers running there also while our client is still running on our local machine.

On cheetah :

  • install uWSGI as described earlier
  • install Consul as described earlier

Run a Consul agent (no need of a server) and tell him to work with your already running consul server on your box (drakar in my case) :

$ /usr/bin/consul agent -data-dir=/tmp/consul-agent -join drakar -ui-dir=/var/lib/consul/ui -client=0.0.0.0

The -join <your host or IP> is the important part.

 

Now run uWSGI so it starts and registers two new servers on cheetah :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2

 

And check the miracle on your client terminal still running on your local box, the new servers have appeared and will disappear if you stop uwsgi on the cheetah node :

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2001
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2002

Go mad

Check the source code, it’s so simple and efficient you’ll cry ;)

I hope this example has given you some insights and ideas for your current or future application designs !

August 12, 2014
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
HD Daymaker LED Headlamp (August 12, 2014, 05:10 UTC)

2014-08-12-064921_571x472_scrot

Short post to share my experience with the Harley-Davidson Daymaker LED Headlamp.

I came to buy it because I was not satisfied with the standard lamp fitted on my sportster and I guess whoever has to drive by night would feel that unpleasant feeling to not actually be able to properly see what’s going on in front of you.

The LED Headlamp is worth the few hundred bucks it costs if at least for the sake of your own life but furthermore for the incredible improvement from the standard lamp. Don’t hesitate a second just go for it and it’s dead simple to mount yourself !

See the difference (passing lights) :

IMG_20140804_215931

IMG_20140804_224657

Now I feel way safer to drive on unlitten roads.

August 11, 2014
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Releases! (August 11, 2014, 11:44 UTC)

Last we made a huge effort to make a release for every supported branch (and even one that is supposed not to be). Lots of patches to fix some old bugs got backported. I hope you appreciate the dedication.

Libav 0.8.15

We made an extra effort, this branch is supposed to be closed and the code is really ancient!
I went the extra mile and I had to run over all the codebase to fix a security issue properly: you might crash if your get_buffer callback doesn’t validate the frame dimension, that code is provided by the library user (e.g. VLC), so the solution is to wrap the get_buffer callback in a function ff_get_buffer and do the check there. For Libav 9 and following we did already for unrelated reasons, for Libav 0.8 I (actually we since the first patch didn’t cover all usage) had sift through the code and replace all the avctx->get_buffer() with ff_get_buffer().

Libav 9.16

This is a standard security release, backporting from Libav 10 might require some manual retouch since code got cleaned up a lot and some internals are different but it is still less painful than backporting from 11 to 0.8

Libav 10.3

This is a quite easy release, backporting fixes is nearly immediate since Libav 11 doesn’t have radical changes in the core internals and the cleanups can apply to release/10.

Libav 11 alpha1

Libav11 is a major release API compatible with Libav10, that makes transitioning as smooth as possible: you enjoy automatically some under-the-hood changes that required an ABI bump (such as the input mime support to speed up AAC webradio startup time) and if you want you can start using the new API features (such as avresample AVFrame API, av_packet_rescale_ts(), AVColor in AVFrame and so on).

You can help!

Libav 11 will be out within the month and help is welcome to polish it and make sure we do not have rough edges.

Update a downstream project you are using

Many downstreams are still using (and sometimes misusing) the old (Libav9) and ancient (Libav0.8) API. We started writing migration guides to help, we contributed many patches already and the Debian packagers did a great job to take care of their side.

Some patches are just waiting to be forwarded to the downstream or, if the package is orphaned, to your favourite distribution packagers.

Triage our bugzilla

Most of the Libav development happens in the mailing-lists and sometimes
bugs reported over bugzilla get not updated timely. Triaging bugs sometimes take a little of time and helps a lot.

Gentoo Monthly Newsletter: July 2014 (August 11, 2014, 00:00 UTC)

Gentoo News

Trustee Election Results

The two open seats for the Gentoo Trustees for the 2014-2016 term will be:

  • Alec Warner (antarus) First Term
  • Roy Bamford (neddyseagoon) Fourth Term

Since there were only two nominees for the two seats up for election, there was no official election. They were appointed uncontested.

Council Election Results

The Gentoo Council for the 2014-2015 term will be:

  • Anthony G. Basile (blueness)
  • Ulrich Müller (ulm)
  • Andreas K. Hüttel (dilfridge)
  • Richard Freeman (rich0)
  • William Hubbs (williamh)
  • Donnie Berkholz (dberkholz)
  • Tim Harder (radhermit)

Official announcement here.

Gentoo Developer Moves

Summary

Gentoo is made up of 242 active developers, of which 43 are currently away.
Gentoo has recruited a total of 803 developers since its inception.

Changes

The following developers have recently changed roles:

  • Projects:
    • mgorny joined Portage
    • k_f joined Gentoo-keys
    • zlogene joined Proxy maintainers
    • civil joined Qt
    • pesa replaced pinkbyte as Qt lead
    • TomWij removed himself from Bug-wranglers
    • Gentoo sound migrated to wiki
    • Artwork migrated to wiki
    • Desktop-util migrated to wiki
    • Accessibility migrated to wiki
    • Enlightenment migrated to wiki
  • Herds:
    • eselect herd was added
    • zlogene joined s390
    • twitch153 joined tools-portage
    • pinkbyte left leechcraft
    • k_f joined crypto

Additions

The following developers have recently joined the project:

  • Xavier Miller (xaviermiller)
  • Patrice Clement (monsieurp)
  • Amy Winston (amynka)
  • Kristian Fiskerstrand (k_f)

Returning Dev

  • Tom Gall (tgall)

Moves

The following developers recently left the Gentoo project:
None this month

Portage

This section summarizes the current state of the portage tree.

Architectures 45
Categories 162
Packages 17595
Ebuilds 37628
Architecture Stable Testing Total % of Packages
alpha 3658 561 4219 23.98%
amd64 10863 6239 17102 97.20%
amd64-fbsd 0 1577 1577 8.96%
arm 2681 1743 4424 25.14%
arm64 559 32 591 3.36%
hppa 3061 482 3543 20.14%
ia64 3189 612 3801 21.60%
m68k 618 87 705 4.01%
mips 0 2402 2402 13.65%
ppc 6838 2353 9191 52.24%
ppc64 4326 866 5192 29.51%
s390 1477 331 1808 10.28%
sh 1670 403 2073 11.78%
sparc 4114 898 5012 28.49%
sparc-fbsd 0 317 317 1.80%
x86 11535 5288 16823 95.61%
x86-fbsd 0 3237 3237 18.40%

gmn-portage-stats-2014-08

Security

Package Removals/Additions

Removals

Package Developer Date
perl-core/Class-ISA dilfridge 05 Jul 2014
dev-python/argparse mgorny 06 Jul 2014
dev-python/ordereddict mgorny 06 Jul 2014
perl-core/Filter dilfridge 07 Jul 2014
app-text/qgoogletranslator grozin 09 Jul 2014
dev-lisp/openmcl grozin 09 Jul 2014
dev-lisp/openmcl-build-tools grozin 09 Jul 2014
net-libs/cyassl blueness 15 Jul 2014
dev-ruby/text-format graaff 18 Jul 2014
dev-ruby/jruby-debug-base graaff 18 Jul 2014
games-util/rubygfe graaff 18 Jul 2014
perl-core/PodParser dilfridge 20 Jul 2014
virtual/perl-PodParser dilfridge 21 Jul 2014
perl-core/digest-base dilfridge 22 Jul 2014
virtual/perl-digest-base dilfridge 22 Jul 2014
perl-core/i18n-langtags dilfridge 22 Jul 2014
virtual/perl-i18n-langtags dilfridge 22 Jul 2014
perl-core/locale-maketext dilfridge 23 Jul 2014
virtual/perl-locale-maketext dilfridge 23 Jul 2014
perl-core/net-ping dilfridge 23 Jul 2014
virtual/perl-net-ping dilfridge 23 Jul 2014
virtual/perl-Switch dilfridge 25 Jul 2014
perl-core/Switch dilfridge 25 Jul 2014
x11-misc/keytouch pacho 27 Jul 2014
x11-misc/keytouch-editor pacho 27 Jul 2014
media-video/y4mscaler pacho 27 Jul 2014
dev-python/manifestdestiny pacho 27 Jul 2014
dev-cpp/libsexymm pacho 27 Jul 2014

Additions

Package Developer Date
www-client/vimb radhermit 01 Jul 2014
dev-util/libsparse jauhien 01 Jul 2014
dev-python/docker-py chutzpah 01 Jul 2014
dev-util/ext4_utils jauhien 01 Jul 2014
dev-haskell/base16-bytestring gienah 02 Jul 2014
dev-haskell/boxes gienah 02 Jul 2014
dev-haskell/chell gienah 02 Jul 2014
dev-haskell/conduit-extra gienah 02 Jul 2014
dev-haskell/cryptohash-conduit gienah 02 Jul 2014
dev-haskell/ekg-core gienah 02 Jul 2014
dev-haskell/equivalence gienah 02 Jul 2014
dev-haskell/hastache gienah 02 Jul 2014
dev-haskell/options gienah 02 Jul 2014
dev-haskell/patience gienah 02 Jul 2014
dev-haskell/prelude-extras gienah 02 Jul 2014
dev-haskell/tf-random gienah 02 Jul 2014
dev-haskell/quickcheck-instances gienah 02 Jul 2014
dev-haskell/streaming-commons gienah 02 Jul 2014
dev-haskell/vector-th-unbox gienah 02 Jul 2014
dev-haskell/tasty-th gienah 02 Jul 2014
dev-haskell/dlist-instances gienah 02 Jul 2014
dev-haskell/temporary-rc gienah 02 Jul 2014
dev-haskell/stmonadtrans gienah 02 Jul 2014
dev-haskell/data-hash gienah 02 Jul 2014
dev-haskell/yesod-auth-hashdb gienah 02 Jul 2014
sci-mathematics/agda-lib-ffi gienah 02 Jul 2014
dev-haskell/lifted-async gienah 02 Jul 2014
dev-haskell/wai-conduit gienah 02 Jul 2014
dev-haskell/shelly gienah 02 Jul 2014
dev-haskell/chell-quickcheck gienah 03 Jul 2014
dev-haskell/tasty-ant-xml gienah 03 Jul 2014
dev-haskell/lcs gienah 03 Jul 2014
dev-haskell/tasty-golden gienah 03 Jul 2014
sec-policy/selinux-tcsd swift 04 Jul 2014
dev-perl/Class-ISA dilfridge 05 Jul 2014
net-wireless/gqrx zerochaos 06 Jul 2014
dev-perl/Filter dilfridge 07 Jul 2014
app-misc/abduco xmw 10 Jul 2014
virtual/perl-Math-BigRat dilfridge 10 Jul 2014
virtual/perl-bignum dilfridge 10 Jul 2014
dev-perl/Net-Subnet chainsaw 11 Jul 2014
dev-java/opencsv ercpe 11 Jul 2014
dev-java/trident ercpe 11 Jul 2014
dev-java/htmlparser-org ercpe 11 Jul 2014
dev-java/texhyphj ercpe 12 Jul 2014
dev-util/vmtouch dlan 12 Jul 2014
sys-block/megactl robbat2 14 Jul 2014
dev-python/fexpect jlec 14 Jul 2014
mail-filter/postfwd mschiff 15 Jul 2014
dev-python/wheel djc 15 Jul 2014
dev-ruby/celluloid-io mrueg 15 Jul 2014
sys-process/tiptop patrick 16 Jul 2014
dev-ruby/meterpreter_bins zerochaos 17 Jul 2014
sys-power/thermald dlan 17 Jul 2014
net-analyzer/check_mk dlan 17 Jul 2014
app-admin/fleet alunduil 19 Jul 2014
perl-core/Pod-Parser dilfridge 20 Jul 2014
virtual/perl-Pod-Parser dilfridge 21 Jul 2014
sci-libs/libcerf ottxor 21 Jul 2014
games-fps/enemy-territory-omnibot ottxor 22 Jul 2014
dev-libs/libflatarray slis 22 Jul 2014
perl-core/Digest dilfridge 22 Jul 2014
virtual/perl-Digest dilfridge 22 Jul 2014
net-libs/stem mrueg 22 Jul 2014
perl-core/I18N-LangTags dilfridge 22 Jul 2014
virtual/perl-I18N-LangTags dilfridge 22 Jul 2014
perl-core/Locale-Maketext dilfridge 22 Jul 2014
virtual/perl-Locale-Maketext dilfridge 23 Jul 2014
perl-core/Net-Ping dilfridge 23 Jul 2014
virtual/perl-Net-Ping dilfridge 23 Jul 2014
dev-libs/libbson ultrabug 23 Jul 2014
sci-libs/silo slis 24 Jul 2014
dev-python/pgpdump jlec 24 Jul 2014
net-libs/libasr zx2c4 25 Jul 2014
dev-libs/npth zx2c4 25 Jul 2014
net-wireless/bladerf-firmware zerochaos 25 Jul 2014
net-wireless/bladerf-fpga zerochaos 25 Jul 2014
net-wireless/bladerf zerochaos 25 Jul 2014
sci-libs/cgnslib slis 25 Jul 2014
sci-visualization/visit slis 25 Jul 2014
dev-perl/Switch dilfridge 25 Jul 2014
dev-util/objconv slyfox 28 Jul 2014
app-crypt/monkeysign k_f 29 Jul 2014
virtual/bitcoin-leveldb blueness 29 Jul 2014
dev-db/percona-server robbat2 29 Jul 2014
sys-cluster/galera robbat2 30 Jul 2014
dev-db/mariadb-galera robbat2 30 Jul 2014
net-im/corebird dlan 30 Jul 2014
dev-libs/libpfm slis 31 Jul 2014
dev-perl/ExtUtils-Config civil 31 Jul 2014
dev-libs/papi slis 31 Jul 2014
dev-perl/ExtUtils-Helpers civil 31 Jul 2014
sys-cluster/hpx slis 31 Jul 2014
dev-perl/ExtUtils-InstallPaths civil 31 Jul 2014
dev-perl/Module-Build-Tiny civil 31 Jul 2014
www-plugins/pipelight ryao 31 Jul 2014

Bugzilla

The Gentoo community uses Bugzilla to record and track bugs, notifications, suggestions and other interactions with the development team.

Activity

The following tables and charts summarize the activity on Bugzilla between 01 July 2014 and 31 July 2014. Not fixed means bugs that were resolved as NEEDINFO, WONTFIX, CANTFIX, INVALID or UPSTREAM.
gmn-activity-2014-07

Bug Activity Number
New 1405
Closed 958
Not fixed 164
Duplicates 180
Total 5912
Blocker 5
Critical 19
Major 69

Closed bug ranking

The following table outlines the teams and developers with the most bugs resolved during this period

Rank Team/Developer Bug Count
1 Gentoo KDE team 41
2 Gentoo Security 38
3 Java team 29
4 Gentoo's Team for Core System packages 28
5 Gentoo Linux Gnome Desktop Team 24
6 Gentoo Games 24
7 Netmon Herd 23
8 Qt Bug Alias 22
9 Perl Devs @ Gentoo 22
10 Others 706

gmn-closed-2014-07

Assigned bug ranking

The developers and teams who have been assigned the most bugs during this period are as follows.

Rank Team/Developer Bug Count
1 Gentoo Linux bug wranglers 85
2 Gentoo Linux Gnome Desktop Team 64
3 Gentoo Security 56
4 Gentoo's Team for Core System packages 53
5 Julian Ospald (hasufell) 48
6 Netmon Herd 47
7 Gentoo KDE team 47
8 Python Gentoo Team 31
9 media-video herd 30
10 Others 943

gmn-opened-2014-07

Tip of the month

(by Sven Vermeulen)
Launching commands in background once (instead of scheduled through cron)

  • Have sys-process/at installed.
  • Have /etc/init.d/atd started.

Use things like:
~$ echo "egencache --update --repo=gentoo --jobs=4" | at now + 10 minutes

Heard in the community

Send us your favorite Gentoo script or tip at gmn@gentoo.org

Getting Involved?

Interested in helping out? The GMN relies on volunteers and members of the community for content every month. If you are interested in writing for the GMN or thinking of another way to contribute, please send an e-mail to gmn@gentoo.org.

Comments or Suggestions?

Please head over to this forum post.

August 09, 2014
Rafael Goncalves Martins a.k.a. rafaelmartins (homepage, bugs)
Introducing pyoembed (August 09, 2014, 21:46 UTC)

Warning: This is a (very) delayed announcement! ;-)

oEmbed is an open standard for embedded content. It allows users to embed some resource, like a picture or a video, in a web page using only the resource URL, without knowing the details of how to embed the resource in a web page.

oEmbed isn't new stuff. It was created around 2008, and despite not being widely supported by content providers, it is supported by some big players, like YouTube, Vimeo, Flickr and Instagram, making its usage highly viable.

To support the oEmbed standard, the content provider just needs to provide a simple API endpoint, that receives an URL and a few other parameters, like the maximum allowed height/width, and returns a JSON or XML object, with ready-to-use embeddable code.

The content provider API endpoint can be previously known by the oEmbed client, or auto-discovered using some meta tags added to the resource's HTML page. This is the point where the standard isn't precise enough: not all of the providers support auto-discovering of the API endpoint, neither all of the providers are properly listed on the oEmbed specification. Proper oEmbed clients should try both approaches, looking for known providers first, falling back to auto-discovered endpoints, if possible.

Each of the Python libraries for oEmbed decided to follow one of the mentioned approaches, without caring about the other one, failing to support relevant providers. And this is the reason why I decided to start writing pyoembed!

pyoembed is a simple and easy to use implementation of the oEmbed standard for Python, that supports both auto-discovered and explicitly defined providers, supporting most (if not all) the relevant providers.

pyoembed's architecture makes it easy to add new providers and supports most of the existing providers out of the box.

To install it, just type:

$ pip install pyoembed

Gentoo users can install it from gentoo-x86:

# emerge -av pyoembed

pyoembed is developed and managed using Github, the repository is publicly available:

https://github.com/rafaelmartins/pyoembed

A Jenkins instance runs the unit tests and the integration tests automatically, you can check the results here:

https://ci.rgm.io/view/pyoembed/

The integration tests are supposed to fail from time to time, because they rely on external urls, that may be unavailable while the tests are running.

pyoembed is released under a 3 clause BSD license.

Enjoy!

Sven Vermeulen a.k.a. swift (homepage, bugs)
Some changes under the hood (August 09, 2014, 19:45 UTC)

In between conferences, technical writing jobs and traveling, we did a few changes under the hood for SELinux in Gentoo.

First of all, new policies are bumped and also stabilized (2.20130411-r3 is now stable, 2.20130411-r5 is ~arch). These have a few updates (mergers from upstream), and r5 also has preliminary support for tmpfiles (at least the OpenRC implementation of it), which is made part of the selinux-base-policy package.

The ebuilds to support new policy releases now are relatively simple copies of the live ebuilds (which always contain the latest policies) so that bumping (either by me or other developers) is easy enough. There’s also a release script in our policy repository which tags the right git commit (the point at which the release is made), creates the necessary patches, uploads them, etc.

One of the changes made is to “drop” the BASEPOL variable. In the past, BASEPOL was a variable inside the ebuilds that pointed to the right patchset (and base policy) as we initially supported policy modules of different base releases. However, that was a mistake and we quickly moved to bumping all policies with every releaes, but kept the BASEPOL variable in it. Now, BASEPOL is “just” the ${PVR} value of the ebuild so no longer needs to be provided. In the future, I’ll probably remove BASEPOL from the internal eclass and the selinux-base* packages as well.

A more important change to the eclass is support for the SELINUX_GIT_REPO and SELINUX_GIT_BRANCH variables (for live ebuilds, i.e. those with the 9999 version). If set, then they pull from the mentioned repository (and branch) instead of the default hardened-refpolicy.git repository. This allows for developers to do some testing on a different branch easily, or for other users to use their own policy repository while still enjoying the SELinux integration support in Gentoo through the sec-policy/* packages.

Finally, I wrote up a first attempt at our coding style, heavily based on the coding style from the reference policy of course (as our policy is still following this upstream project). This should allow the team to work better together and to decide on namings autonomously (instead of hours of discussing and settling for something as silly as an interface or boolean name ;-)

Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)
unpaper and libav status update (August 09, 2014, 11:46 UTC)

The other day I wrote about unpaper and the fact that I was working on making it use libav for file input. I have now finished converting unpaper (in a branch) so that it does not use its own image structure, but rather the same AVFrame structure that libav uses internally and externally. This meant not only supporting stripes, but using the libav allocation functions and pixel formats.

This also enabled me to use libav for file output as well as input. While for the input I decided to add support for formats that unpaper did not read before, for output at the moment I'm sticking with the same formats as before. Mostly because the one type of output file I'd like to support is not currently supported by libav properly, so it'll take me quite a bit longer to be able to use it. For the curious, the format I'm referring to is multipage TIFF. Right now libav only supports single-page TIFF and it does not support JPEG-compressed TIFF images, so there.

Originally, I planned to drop compatibility with previous unpaper version, mostly because to drop the internal structure I was going to lose the input format information for 1-bit black and white images. At the end I was actually able to reimplement the same feature in a different way, and so I restored that support. The only compatibility issue right now is that the -depth parameter is no longer present, mostly because it and -type constrained the same value (the output format).

To reintroduce the -depth parameter, I want to support 16-bit gray. Unfortunately to do so I need to make more fundamental changes to the code, as right now it expects to be able to get the full value at most at 24 bit — and I'm not sure how to scale a 16-bit grayscale to 24-bit RGB and maintain proper values.

While I had to add almost as much code to support the libav formats and their conversion as there was there to load the files, I think this is still a net win. The first point is that there is no format parsing code in unpaper, which means that as long as the pixel format is something that I can process, any file that libav supports now or will support in the future will do. Then there is the fact that I ended up making the code "less smart" by removing codepath optimizations such as "input and output sizes match, so I won't be touching it, instead I'll copy one structure on top of the other", which means that yes, I probably lost some performance, but I also gained some sanity. The code was horribly complicated before.

Unfortunately, as I said in the previous post, there are a couple of features that I would have preferred if they were implemented in libav, as that would mean they'd be kept optimized without me having to bother with assembly or intrinsics. Namely pixel format conversion (which should be part of the proposed libavscale, still not reified), and drawing primitives, including bitblitting. I think part of this is actually implemented within libavfilter but as far as I know it's not exposed for other software to use. Having optimized blitting, especially "copy this area of the image over to that other image" would be definitely useful, but it's not a necessary condition for me to release the current state of the code.

So current work in progress is to support grayscale TIFF files (PAL8 pixel format), and then I'll probably turn to libav and try to implement JPEG-encoded TIFF files, if I can find the time and motivation to do so. What I'm afraid of is having to write conversion functions between YUV and RGB, I really don't look forward to that. In the mean time, I'll keep playing Tales of Graces f because I love those kind of games.

Also, for those who're curious, the development of this version of unpaper is done fully on my ZenBook — I note this because it's the first time I use a low-power device to work on a project that actually requires some processing power to build, but the results are not bad at all. I only had to make sure I had swap enabled: 4GB of RAM are no longer enough to have Chrome open with a dozen tabs, and a compiler in the background.

August 07, 2014
Paweł Hajdan, Jr. a.k.a. phajdan.jr (homepage, bugs)
Can your distro compile Chromium? (August 07, 2014, 07:20 UTC)

Chromium is moving towards using C++11. Even more, it's going to require either gcc-4.8 or clang.

Distros like Ubuntu, Mageia, Fedora, openSUSE, Arch, CentOS, and Slackware are already using gcc-4.8 or later is their latest stable release.

On the other hand, Debian Wheezy (7.0) has gcc-4.7.2. Gentoo is using gcc-4.7.3 in stable.

I started a thread on gentoo-dev, gcc-4.8 may be needed in stable for www-client/chromium-38.x. There is a tracker for gcc-4.8 stabilization, bug #516152. There is also gcc-4.8 porting tracker, bug #461954.

Please consider testing gcc-4.8 on your stable Gentoo system, and file bugs for any package that fails to compile or needs to have a newer version stabilized to work with new gcc. I have recompiled all packages, the kernel, and GRUB without problems.

The title of this post is deliberately a bit similar to my earlier post Is your distro fast enough for Chromium? This browser project is pushing a lot towards shorter release cycles and latest software. I consider that a good thing. Now we just need to keep up with the updates, and any help is welcome.

Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
googlecode.com, or no tarballs for you (August 07, 2014, 02:45 UTC)

I'm almost amused, see this bug

So when I fetched it earlier the tarball had size 207200 bytes
Most of europe apparently gets a tarball of size 207135 bytes
When I download now again I get a tarball of size 206989 bytes

So I have to assume that googlecode now follows githerp in their tradition of being useless for code hosting. Is it really that hard to generate a consistent tarball once, and then mirror it?
Maybe I should build my own codehosting just to understand why this is apparently impossible ...

Jeremy Olexa a.k.a. darkside (homepage, bugs)
What’s new? (August 07, 2014, 01:17 UTC)

Ahem, let me dust this this off…

For those keeping track at home, it has been over 7 months since writing on this thing. Yup, new job, new car, new apartment after I got back. That was fun, and “settling” in again has kept me busy. I’ve also been enjoying the [short] summer that we have.

The most common question that people ask me now is “When are you leaving again?” – I guess there must be something in my eyes when I tell the travel story…ha. Nothing planned.

As far as tech goes, I’ve been digging into Chef for my IT automation needs. I simply can’t imagine a workplace without automation these days. I would show some github stats here but, (said every Ops engineer that I know,) most everything is behind private repo(s). I’m learning new technologies I haven’t used before and wearing many hats at a startup. I know the breadth of skills can only help in the long run. I haven’t worked on Gentoo Linux in awhile. I’m trying to find something there that interests me but after your tech belongings have been commoditized/optimized for lightweight travel, motivation is lacking. Keeping up with emerging tech is still fun, though.

August 06, 2014
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

I am no stranger to Indian food, as it is among my favourite types of cuisine (along with Thai and Vietnamese). Having been back in the Saint Louis area for two years now, I have tried many different Indian restaurants, but have been disappointed for one reason or another (price, variety of regional dishes, a lack of distinct flavour profiles, et cetera). Please don’t misunderstand me; there are some good, and even some great Indian places in and around Saint Louis, but they have all somehow fallen a bit short. For instance, here are some such places:

  • India’s Rasoi – great, but expensive, and no buffet
  • Haveli – pretty good buffet, but lacking some variety
  • India’s Kitchen – decent buffet, but inconsistent; nothing stands out
  • Copper Chimney – pretty good, but not all that many options
  • Saffron – decent buffet, but again, not all that many regional options

That list is in no way exhaustive, but I think that the theme will be evident—they’re good, but not “stop you in your tracks” good. Having lived in some regions of the country that have a plethora of exceptional Indian restaurants, I was constantly on a mission to find The Best Indian Restaurant in Saint Louis! My search has yielded a winner: Peshwa Indian Restaurant.

I went with my dearest friend and fellow foodie, Debbie, very shortly after owner and executive Shweta Marathe opened the doors to her wonderful new eatery. As of the time of this blog, we have been back six times in just a few short weeks! Why go back so often (other than the obvious reason of the food is incredible)? Variety. Peshwa constantly has new dishes on the buffet, all stemming from the myriad regions of India. Ms. Marathe brings her unique take on these dishes, and spices them up (pun intended) with influences from her native region outside Pune, which is near the Western coast of India (southeast of Mumbai).


Vada with tamarind Chutney
(Click to enlarge)

This most recent time, we started with some Vada with Tamarind chutney. Vada are these wonderful little doughnut-like delicacies from South India, and are typically made from Urad dal and gram flour. I can’t say if these ones were made primarily with dal or lentils, but they were delicious. As with many dishes (not just from India), the sauces make or break them. The tamarind chutney at Peshwa is the best that I’ve ever had. It has the sweetness (from the dates) that I’ve come to love but haven’t found at other places.

Other appetisers that we’ve had in the past are Idli, which are typically eaten as a dense breakfast food accompanied by a coconut chutney. When I was discussing with Shweta how much I enjoyed these cakes made from Urad dal, I mentioned coconut chutney. She educated me and let me know that they are also eaten with Sambar. I tried them that way, and it was a completely different experience! At the same time, she was back in the kitchen whipping up some coconut chutney (now THAT’S service)!


Chicken Tikka Masala, Vegetable Korma, Naan, and rice
(Click to enlarge)

For entrées, Peshwa offers far too many to list, including some wonderful vegetarian and vegan dishes. The first few times that we went, one of the primary chicken offerings was Butter Chicken, which is great, but not my favourite. That being said, this was outstanding (not overly oily, like it has been at some other places). After talking with Shweta, she agreed to make Chicken Tikka Masala for me at some point (since it is one of my absolute favourites). I used to think that India’s Rasoi had the best in the area, but it has been surpassed in my opinion. At Peshwa, there is not as much sauce, but what is there is infinitely flavourful. The pieces of chicken are so tender that one doesn’t need a knife at all.

Typically, Vegetable Korma is enjoyable, but not something that jumps off the buffet line onto my plate as readily as some other choices. At Peshwa, though, I believe that it is one of the absolute best dishes available. It is creamy and has a flavour profile that is both subtle and complex.

Many other main dishes are available on the buffet as well. You can find staples like Tandoori Chicken, various styles of Biryani, Vindaloo, as well as some lesser-known dishes and even Indian Chinese dishes, which are really something special!


My own mixture of Pineapple Sheera and Kheer rice pudding
(Click to enlarge)

Now, after indulging in those wonderfully complex and sometimes spice-filled entrées, one wants (or even needs) some desserts to cool down the palette. At Peshwa, there are usually two or three desserts available, and they’re constantly being rotated out for different ones. One of the recent times that we went, I was excited to see that two of my favourites (Pineapple Sheera, and Kheer) were both available at the same time. One thing that I love to do, (even though it’s not very traditional), is to mix the two together. I really enjoy the juxtaposition of the warm Sheera and the cool Kheer, as well as the combination of two different textures. Now, Kheer comes in many different varieties, and I have had two of them at Pesha. This particular day, it was the Kheer that is more like a rice pudding with shaved almonds. One previous time, another outstanding dessert was on the menu: Gulab Jamun, which can be found most often in Western India.

I would be amiss if I neglected to mention one special dessert that I’ve only found at Peshwa—the Mango Mastani. This refreshing flavour explosion is native to Pune and surrounding areas, and is made from mango (duh), cold whole milk, sugar, ice cubes, and mango kulfi. It is basically like a mango shake / float with a big scoop of mango kulfi (ice cream-like) in it. Nothing can prepare you for the immense flavour of this outstanding dessert. The only problem that you will have (if you’re like me) is leaving room for it at the end of an otherwise excellent meal.

If you’ve stuck with me throughout this entire review, you’ll easily know that I think VERY highly of Peshwa. Not only has every dish I’ve had there been incredible, but the service is great as well. Deb and I keep joking that one day we’ll find a dish that Shweta and her staff don’t do well, but we’ve yet to find it. If I had to come up with a fault of the restaurant, I would have to be nitpicky to an extreme. Doing so, though, I would say that it would be nice to have some more ice in the water, but I understand this is a typically Western idea.

Do yourself a huge favour, and check out Peshwa Indian Restaurant at:
10633 Page Avenue (click for directions)
Suite B
Saint Louis, MO 63132

As of this writing, they are open from 11:30 until 20:30 (8:30 PM) every day but Tuesday.

Cheers, and happy eating!

|:| Zach |:|

Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)

As I noted earlier, I've been doing some more housecleaning of bad HTTP crawlers and feed readers. While it matters very little for my and my blog (I don't pay for bandwidth), I find it's a good exercise and, since I do publish my ModSecurity rules, it is a public service for many.

For those who think that I may be losing real readership in this, the number of visits on my site as seen by Analytics increased (because of me sharing the links to that post over to Twitter and G+, as well as in the GitHub issues and the complaint email I sent to the FeedMyInbox guys), yet the daily traffic was cut in half. I think this is what is called a win-win.

But one thing that became clear from both AWSstats and Analytics is that there was one more crawler that I did not stop yet. The crawler name is Semalt, and I'm not doing them the favour of linking to their website. Those of you who follow me on twitter have probably seen what they categorized as "free PR" for them, while I was ranting them up. I defined them a cancer for the Internet, I then realized that the right categorization would be bacteria.

If you look around, you'll find unflattering reviews and multiple instructions to remove them from your website.

Funnily, once I tweeted about my commit, one of their people, who I assume is in their PR department rather than engineering for the blatant stupidity of their answers, told me that it's "easy" to opt-out of their scanner.. you just have to go on their website and tell them your websites! Sure, sounds like a plan, right?

But why on earth am I spending my time attacking one particular company that, to be honest, is not wasting that much of my bandwidth to begin with? Well, as you can imagine from me comparing them to shigella bacteria, I do have a problem with their business idea. And given that on twitter they even missed completely my point (when I pointed out the three spammy techniques they use, their answer was "people don't complain about Google or Bing" — well, yes, neither of the two use any of their spammy techniques!), it'll be difficult for me to consider them as mistaken. They are doing this on purpose.

Let's start with the technicalities, although that's not why I noticed them to begin with. As I said earlier, their way to "opt out" from their services is to go to their website and fill in a form. They completely ignore robots.txt, they don't even fetch it. And given this is an automated crawler, that's bad enough.

The second is that they don't advertise themselves in the User-Agent header. Instead all their fetches report Chrome/35 — and given that they can pass through my ruleset, they probably use a real browser with something like WebDriver. So you have no real way to identify their requests among a number of others, which is not how a good crawler should operate.

The third and most important point is the reason why I consider them just spammers, and so seem others, given the links I posted earlier. Instead of using the user agent field to advertise themselves, they subvert the Referer header. Which means that all their requests, even those that have been 301'd and 302'd around, will report their website as referrer. And if you know how AWStats works, you know that it doesn't take that many crawls for them to be one of the "top referrers" for your website, and thus appear prominently in your stats, whether they are public or not.

At this point it could be easy to say that they are clueless and are not doing this on purpose, but then there is the other important part. Their crawler executes JavaScript, which means that it gets tracked by Google Analytics, too! Analytics has no access to the server logs, so for it to display the referrer as shown by people looking to filter it out, it has to make an effort. Again, this could easily be a mistake, given that they are using something like WebDriver, right?

The problem is that whatever they use, it does not fetch either images or CSS. But it does fetch the Analytics javascript and execute it, as I said. And the only reason I can think for them to want to do so, is to spam the referrer list in there as well.

As their twitter person thanked me for my "free PR" for them, I wanted to expand it further on it, with the hope that people will learn to know them. And to avoid them. My ModSecurity ruleset as I said already is set up to filter them out, other solutions for those who don't want to use ModSecurity are linked above.

August 05, 2014
Hanno Böck a.k.a. hanno (homepage, bugs)
Las Vegas (August 05, 2014, 05:39 UTC)

Excalibur hotel
My hotel looks like a Disneyland castle - just much larger.
I am a regular author for the German IT news page Golem.de. Earlier this year they asked me if I wanted to report from the Black Hat and Def Con conferences in Las Vegas. The conferences will start tomorrow. As I don't want to fly half across the globe for a few days of IT security conferences I decided to spend some more time. So I spend the last couple of days in Las Vegas and will spend some time after the conferences travelling around. A couple of people asked me to blog a bit and post some pictures, so here we go.

Las Vegas is probably a place I would've never visited on its own. I consider myself a rationalist person and therefore I see gambling mostly as an illogical pursuit. In the end your chances of winning are minimal because otherwise the business wouldn't work. I hadn't imagined how huge the casino business in Las Vegas is. Large parts of the city are just one large casino after another - and it doesn't stop there, because a couple of cities around Vegas literally are made of casinos.

Beside seeing some of the usual tourist attractions (Hoover Dam, Lake Mead), I spend the last couple of days also finding out that there are some interesting solar energy projects nearby. Also a large Star Trek convention> was happening the past days where I attended on the last day.

Nintendo test cardrige
A Nintendo test cardrige at A Gamer's Paradise
If you are ever in Vegas and have the slightest interest in retro gaming I suggest to visit A Gamer's Paradise. It is a shop for used video games, but apart from that it is also a showcase for a whole range of old and partly very exotic video gaming equipment, including things I've never seen before. It also has some playable old consoles. Right beside of it is the Pinball Hall of Fame, which is also a nice place. So you can visit two worthwhile retro gaming related places in one go.

Pictures from Las Vegas
Pictures from A Gamer's Paradise
Pictures from Pinball Hall of Fame

August 04, 2014
Sebastian Pipping a.k.a. sping (homepage, bugs)

(Last edit 2014-08-31)

I wrote a tool that takes the board setup from a WXF file (example below) and produces an SVG image visualizing that setup.
Different themes for board and pieces are supported (including your own), the gap between pieces can be adjusted, output width can be adjusted, too.
It’s called xiangqi-setup.

Internally, the tool takes the an SVG file of the board, places piece SVGs at the right places and saves the result.
The tool uses svgutils by Bartosz Telenczu. I’m very happy he made that available as free software. My tool is free software (licensed under GNU AGPL 3.0 or later) too, of course.

If you want to imitate the style of Chinese end-game books, you could go with these themes:


(Image licensed under CC0 1.0 Universal: Public Domain Dedication)
If you want to imitate LaTeX xq 0.3 style (with added flexibility), you could go with this:


(Image licensed under CC0 1.0 Universal: Public Domain Dedication)
If you would rather go for something more colourful, for screen rather than print, or you want to explicitly imitate the look of PlayOK.com, you could go with this:


(Image licensed under CC0 1.0 Universal: Public Domain Dedication,
piece artwork kindly shared and released by PlayOK)
There is a version of the pieces without shadows (for PDF generation), too.

Latter image was created running

# ./xiangqi-setup \
    --board themes/board/playok_2014_remake/ \
    --pieces themes/pieces/playok_2014_chinese \
    --scale-pieces 1.025 \
    --width-px 400 \
    demo.wxf setup_imitate_playok.svg

This what you pass in: a WXF file (e.g. produced by XieXie by saving with .wxf extension):

FORMAT          WXF
GAME    
RED             ;;;
BLACK           ;;;
DATE            2014-07-16
FEN             4kaer1/4a2c1/2h1e1h2/3Rp1C1p/2C6/5rP2/1pP1P3P/8E/9/1cEAKA1R1 b

START{
}END

As of now, the complete usage of xiangqi-setup is:

# ./xiangqi-setup --help
usage: xiangqi-setup [-h] [--board DIRECTORY] [--pieces DIRECTORY]
                     [--width-px PIXEL] [--width-cm CENTIMETER] [--dpi FLOAT]
                     [--scale-pieces FACTOR] [--debug]
                     INPUT_FILE OUTPUT_FILE

positional arguments:
  INPUT_FILE
  OUTPUT_FILE

optional arguments:
  -h, --help            show this help message and exit
  --board DIRECTORY
  --pieces DIRECTORY
  --width-px PIXEL
  --width-cm CENTIMETER
  --dpi FLOAT
  --scale-pieces FACTOR
  --debug

For themes, these are your options (at the moment):

# find themes -maxdepth 2 -type d | sort
themes
themes/board
themes/board/a4_blank_2cm_margin
themes/board/clean_alpha
themes/board/clean_beta
themes/board/commons_xiangqi_board_2008
themes/board/commons_xiangqi_board_2008_bw_thin
themes/board/latex_xq_remake
themes/board/minimal
themes/board/minimal_chinese
themes/board/minimal_chinese_arabic
themes/board/playok_2014_remake
themes/pieces
themes/pieces/commons_xiangqi_pieces_print_2010
themes/pieces/commons_xiangqi_pieces_print_2010_bw_heavy
themes/pieces/latex_xqlarge_2006_chinese_autotrace
themes/pieces/latex_xqlarge_2006_chinese_potrace
themes/pieces/playok_2014_chinese
themes/pieces/playok_2014_chinese_noshadow
themes/pieces/retro_simple

If none of the existing themes fit your needs, you may create board and/or pieces of your own.
For boards, drawing a custom grid, palace, start markers and border can be done using the xiangqi-board tool. A demonstration of its current options:

# ./xiangqi-board --help
usage: xiangqi-board [-h] [--line-thickness-px FLOAT] [--field-width-px FLOAT]
                     [--field-height-px FLOAT] [--border-thickness-px FLOAT]
                     [--border-gap-width-px FLOAT]
                     [--border-gap-height-px FLOAT] [--cross-width-px FLOAT]
                     [--cross-thickness-px FLOAT] [--cross-gap-px FLOAT]
                     SVG_FILE INI_FILE

positional arguments:
  SVG_FILE
  INI_FILE

optional arguments:
  -h, --help            show this help message and exit
  --line-thickness-px FLOAT
                        Line thickness of square fields in pixel (default: 1)
  --field-width-px FLOAT
                        Width of fields in pixel (default: 53)
  --field-height-px FLOAT
                        Height of fields in pixel (default: 53)
  --border-thickness-px FLOAT
                        Line thickness of border in pixel (default: 2)
  --border-gap-width-px FLOAT
                        Widtn of gap to border in pixel (default: 40)
  --border-gap-height-px FLOAT
                        Height of gap to border in pixel (default: 40)
  --cross-width-px FLOAT
                        Width of starting position cross segments in pixel
                        (default: 10)
  --cross-thickness-px FLOAT
                        Line thickness of starting position cross in pixel
                        (default: 1)
  --cross-gap-px FLOAT  Gap to starting position cross in pixel (default: 4)

For text on the river, the characters are:

  • Chu river: 楚河
  • Han border: 漢界 traditional, 汉界 simplified

On the Open Source font end of things these are your main options to my understanding:

On a side note, in Gentoo Linux look for these packages:

  • Adobe Source Han Sans: media-fonts/source-han-sans (gentoo-zh overlay)
  • AR PL UKai/UMing CN/TW: media-fonts/arphicfonts
  • Google Noto Sans CJK: media-fonts/notofonts (betagarden overlay)
  • Wangfonts: media-fonts/wangfonts (gentoo-zh overlay)
  • WenQuanYi Micro/Zen Hei: media-fonts/wqy-microhei, media-fonts/wqy-zenhei

If you use xiangqi-setup tool to generate images, feel free to drop me a mail, I would be curious to see your results and check out your custom themes. I would not mind a free of copy of your book, either :)

Cheers!

August 03, 2014
Sebastian Pipping a.k.a. sping (homepage, bugs)

One minute version

If you’re buying an English book on Xiangqi do not buy “A Beginners guide to Xiangqi” by Tyler Rea: It’s a ripp-off and does more harm than good.
For something in English, you could go with “Chinese Chess: An Introduction to China’s Ancient Game of Strategy” by H. T. Lau for print or browse www.xqinenglish.com, instead.

Disclaimer

  • First, sorry for the poor picture quality!
  • If I highlighted say two errors in a picture below, it does not mean it’s only two.
  • This review is not meant to be complete: it’s way too much already.

How did I get here?

I ran into this video on YouTube:

While I’ve been playing Xiangqi for quite a while already, I was thinking “that book looks like fun, I’ll buy it just to have a closer look, maybe there’s something in there that I haven’t seen yet, too“. So I did have some expectations.

I started skip-reading through the book and soon stumbled over error after error: to the point of laugh-or-cry. It starts with the cover page, already. Read on for details.

Quick review summary

  • Whole chapters match Internet content 1:1 (mostly xqinenglish.com, also en.wikipedia.org).
  • About 100 pages are wasted to two uncommented example games, one move per page.
  • Of those, the first example game is declared a win when the king is not even in check.
  • Poor teaching (mis-leading, logic errors, no examples where needed)
  • No chapter explaining AXF/WXF (or any) move notation (despite use in the book)
  • Many errors details, spelling, case, punctuation (even on cover page)

The book lacks declaration of an edition or a print date. It does say “Printed in Germany by Amazon Distribution GmbH, Leipzig” at the end.
According to the book’s Amazon page it was published at/by CreateSpace: Self Publishing and Free Distribution for Books, CD, DVD, “an Amazon company” in 2013.

This beginners guide is actually a beginner’s guide. Maybe that’s a typo.

The cover says it loud and clear


Interesting things to spot on the cover:

  • “Writen” should have been “Written” with double “t”
  • On the bottom, Red’s king and elephant use the characters of Black
  • Red’s pawns show characters of Black’s pawns and vice versa

Too bad I noticed all of that after buying.

Whole chapters match Internet content 1:1

So far I have identified these matches with Internet content (some 1:1, some adjusted):

The author of xqinenglish.com confirmed to me via e-mail that use of his content in that book has not been authorized/licened by him.

The two places where I noticed copying first:

At page 170 it reads “term use[d] on this site” rather than “in this book“:

At page 49 the “Basic, commonly used tactics in Xiangqi” is not a complete sentence and does not have a full stop either. That is because in the original that is the title of a section, not a sentence.

On the example games

While showing a complete game move by move could be helpful in teaching, a few things went wrong in general:

  • Moves should be commented, e.g. “Red is attacking piece X to make up for blacks attack against ..” rather than “Black has responded by advancing the horse”.
  • WXF/AXF and algebraic move notation could have been shown: There is plenty of space for that.
  • Since there are no arrows indicating the current move on the board, “finding” the move is much more work than necessary, especially when turning pages.
  • No more than a single move per page is plain waste of paper.
  • Roughly 100 pages for two examples games take more than half the page count.

On the first game in particular: The game ends at move #79 of Red which is commented as:

WINNING Move #79, Red secures Checkmate. Black’s General is it check from Red’s Chariot and Red’s General as part of a Face to face laughing check.

That’s rather surprising since Black is not even in check after Red has moved. Is this a joke?

On the second game: page 167 shows the second last move of the game. Black responds with R9+2 / Ri0-i8. How does that do anything against the threat of C5=6 / Ce5-d5 by Red? What about C2=5 / Cb8-e8 for a proper reply? If we assume a really poor opponent, maybe that deserves mentioning.

Also, let me use the occasion to point out the characters used for Red’s king and advisors. A friend of mine who has been studying English, Chinese and German literature on Xiangqi for years said he has never seen those characters used anywhere in the context of Xiangqi.

The quality of teaching

This page is meant to teach movement of the king:

What I see is:

  • Pictures indicating movement of more than one step at a time
  • Lack of a system: it’s neither where-to-go-from-here nor where-could-he-have-come-from

Up next is the first oddity I noticed very first: How would the two pawns go sideways before crossing the river?

This page is meant to explain movement of the horse. With that little blocking pieces there are quite a few move options missing.

Here the idea seemed to be: show all possible steps a pawn can do. However, there are lots of arrows missing:

This one is really bad: If I hop my cannon right in front of the king with no protection, he eats my cannon and that’s it:

To me, this dry list of checkmates clearly lacks examples:

No chapter on move notation

Despite use of AXF notation in the book, there is no chapter explaining how to read or write that (or any) move notation. Why not?

Many errors at details, spelling, case, punctuation

In a printed book, finding spelling mistakes is expected to be hard. Not so with this book: A few examples.
On page 4 we can see how punctuation is pulled into quotes — always but once:

The word “Xiangqi” can be found in the book as “Xiàngqí“, “Xiànqí” (typo, lacks “g“) and “Xiangqi” (without accents):

Also, uppercase is used at interesting places, e.g.

  • Page 44: “leap frog Perpetually”
  • Page 45: “The Anatomy and structure”
  • Page 39: “to Augment your defense”

to name a few.

That’s all for the moment.

(Update 2014-08-31: The book is “out-of-order” on Amazon.com and .de; as of today, it is sold by a reseller for 32.05 EUR at Amazon.de and for 189.20(!) USD at Amazon.com.)

Anthony Basile a.k.a. blueness (homepage, bugs)

When portage installs a package onto your system, it caches information about that package in a directory at /var/db/pkg/<cat>/<pkg>/, where <cat> is the category (ie ${CATEGORY}) and <pkg> is the package name, version number and revision number (ie. ${P}). This information can then be used at a later time to tell portage information about what’s installed on a system: what packages were installed, what USE flags are set on each package, what CFLAGS were used, etc. Even the ebuild itself is cached so that if it is removed from the tree, and consequently from your system upon `emerge –sync`, you have a local copy in VDB to uninstall or otherwise continue working with the package. If you take look under /var/db/pkg, you’ll find some interesting and some not so interesting files for each <cat>/<pkg>. Among the less interesting are files like DEPEND, RDPENED, FEATURES, IUSE, USE, which just contain the same values as the ebuild variables by the same name. This is redundant because that information is in the ebuild itself which is also cached but it is more readily available since one doesn’t have to re-parse the ebuild to obtain them. More interesting is information gathered about the package as it is installed, like CONTENTS, which contains a list of all the regular files, directories, and sym link which belong to the package, along with their MD5SUM. This list is used to remove files from the system when uninstalling the package. Environment information is also cached, like CBUILD, CHOST, CFLAGS, CXXFLAGS and LDFLAGS which affects the build of compiled packages, and environment.bz2 which contains the entire shell environment that portage ran in, including all shell variables and functions from inherited eclasses. But perhaps the most interesting information, and the most expensive to recalculate is, cached in NEEDED and NEEDED.ELF.2. The later supersedes the former which is only kept for backward compatibility, so let’s just concentrate on NEEDED.ELF.2. Its a list of every ELF object that is installed for a package, along with its ARCH/ABI information, its SONAME if it is a shared object (readelf -d <obj> | grep SONAME, or scanelf -S), any RPATH used to search for its needed shared objects (readelf -d <obj> | grep RPATH, or scanelf -r), and any NEEDED shared objects (the SONAMES of libraries) that it links against (readelf -d <obj> | grep NEEDED or scanelf -n). [1] Unless you’re working with some exotic systems, like an embedded image where everything is statically linked, your user land utilities and applications depend on dynamic linking, meaning that when a process is loaded from the executable on your hard drive, the linker has to make sure that its needed libraries are also loaded and then do some relocation magic to make sure that unresolved symbols in your executable get mapped to appropriate memory locations in the libraries.

The subtleties of linking are beyond the scope of this blog posting [2], but I think its clear from the previous paragraph that one can construct a “directed linkage graph” [3] of dependencies between all the ELF objects on a system. An executable can link to a library which in turn links to another, and so on, usually back to your libc [4]. `readelf -d <obj> | grep NEEDED` only give you the immediate dependencies, but if you follow these through recursively, you’ll get all the needed libraries that an executable needs to run. `ldd <obj>` is a shell script which provides this information, as does ldd.py from the pax-utils package, which also does some pretty indentation to show the depth of the dependency. If this is sounding vaguely familiar, its because portage’s dependency rules “mimic” the underlying linking which is needed at both compile time and at run time. Let’s take an example, curl compiled with polarssl as its SSL backend:

# ldd /usr/bin/curl | grep ssl
        libpolarssl.so.6 => /usr/lib64/libpolarssl.so.6 (0x000003a3d06cd000)
# ldd /usr/lib64/libpolarssl.so.6
        linux-vdso.so.1 (0x0000029c1ae12000)
        libz.so.1 => /lib64/libz.so.1 (0x0000029c1a929000)
        libc.so.6 => /lib64/libc.so.6 (0x0000029c1a56a000)
        /lib64/ld-linux-x86-64.so.2 (0x0000029c1ae13000)

Now let’s see this dependency reflected in the ebuild:

# cat net-misc/curl/curl-7.36.0.ebuild
RDEPEND="
        ...
        ssl? (
                ...
                curl_ssl_polarssl? ( net-libs/polarssl:= app-misc/ca-certificates )
                ...
        )
        ...

Nothing surprising. However, there is one subtlety. What happens if you update polarssl to a version which is not exactly backwards compatible. Then curl which properly linked against the old version of polarssl doesn’t quite work with the new version. This can happen when the library changes its public interface by either adding new functions, removing older ones and/or changing the behavior of existing functions. Usually upstream indicates this change in the library itself by bumping the SONAME:

# readelf -d /usr/lib64/libpolarssl.so.1.3.7 | grep SONAME
0x000000000000000e (SONAME) Library soname: [libpolarssl.so.6]

But how does curl know about the change when emerging an updated version of polarssl? That’s where subslotting comes in. To communicate the reverse dependency, the DEPEND string in curl’s ebuild has := as the slot indicator for polarssl. This means that upgrading polarssl to a new subslot will trigger a recompile of curl:

# emerge =net-libs/polarssl-1.3.8 -vp

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild r U ] net-libs/polarssl-1.3.8:0/7 [1.3.7:0/6] USE="doc sse2 static-libs threads%* zlib -havege -programs {-test}" ABI_X86="(64) (-32) (-x32)" 1,686 kB
[ebuild rR ] net-misc/curl-7.36.0 USE="ipv6 ldap rtmp ssl static-libs threads -adns -idn -kerberos -metalink -ssh {-test}" CURL_SSL="polarssl -axtls -gnutls -nss -openssl" 0 kB

Here the onus is on the downstream maintainer to know when the API breaks backwards compatibility and subslot accordingly. Going through with this build and then checking the new SONAME we find:

# readelf -d /usr/lib/libpolarssl.so.1.3.8 | grep SONAME
0x000000000000000e (SONAME) Library soname: [libpolarssl.so.7]

Aha! Notice the SONAME jumped from .6 for polarssl-1.3.7 to .7 for 1.3.8. Also notice the SONAME version number also follows the subslotting value. I’m sure this was a conscious effort by hasufell and tommyd, the ebuild maintainers, to make life easy.

So I hope my example has shown the importance of tracing forward and reverse linkage between the ELF objects in on a system [5]. Subslotting is relatively new but the need to trace linking has always been there. There was, and still is, revdep-rebuild (from gentoolkit) which uses output from ldd to construct a “directed linkage graph” [6] but is is relatively slow. Unfortunately, it recalculates all the NEEDED.ELF.2 information on the system in order to reconstruct and invert the directed linkage graph. Subslotting has partially obsoleted revdep-rebuild because portage can now track the reverse dependencies, but it has not completely obsoleted it. revdep-rebuild falls back on the SONAMEs in the shared objects themselves — an error here is an upstream error in which the maintainers of the library overlooked updating the value of CURRENT in the build system, usually in a line of some Makefile.am that looks like

LDFLAGS += -version-info $(CURRENT):$(REVISION):$(AGE)

But an error in subslotting is an downstream error where the maintainers didn’t properly subslot their package and any dependencies to reflect upstream’s changing API. So in some ways, these tools complement each other.

Now we come to the real point of the blog: there is no reason for revdep-rebuild to run ldd on every ELF object on the system when it can obtain that information from VDB. This doesn’t save time on inverting the directed graph, but it does save time on running ldd (effectively /lib64/ld-linux-x86-64.so.2 –list) on every ELF object in the system. So guess what the python version does, revdep-rebuild.py? You guessed it, it uses VDB information which is exported by portage via something like

import portage
vardb = portage.db[portage.root]["vartree"].dbapi

So what’s the difference in time? On my system right now, we’re looking at a difference between approximately 5 minutes for revdep-rebuild versus about 20 seconds for revdep-rebuild.py. [7] Since this information is gathered at build time, there is no reason for any Package Management System (PMS) to not export it via some standarized API. portage does so in an awkward fashion but it does export it. paludis does not export NEEDED.ELF.2 although it does export other VDB stuff. I can’t speak to future PMS’s but I don’t see why they should not be held to a standard.

Above I argued that exporting VDB is useful for utilities that maintain consistency between executibles and the shared objects that they consume. I suspect one could counter-argue that it doesn’t need to be exported because “revdep-rebuild” can be made part of portage or whatever your PMS, but I hope my next point will show that exporting NEEDED.ELF.2 information has other uses besides “consistant linking”. So a stronger point is that, not only should PMS export this information, but that it should provide some well documented API for use by other tools. It would be nice for every PMS to have the same API, preferably via python bindings, but as long as it is well documented, it will be useful. (Eg. webapp-config supports both portage and paludis. WebappConfig/wrapper.py has a simple little switch between “import portage; ... portage.settings['CONFIG_PROTECT'] ... ” and “cave print-id-environment-variable -b --format '%%v\n' --variable-name CONFIG_PROTECT %s/%s ...“.)

So besides consistent linking, what else could make use of NEEDED.ELF.2? In the world of Hardened Gentoo, to increase security, a PaX-patched kernel holds processes to much higher standards with respect to their use of memory. [8] Unfortunately, this breaks some packages which want to implement insecure methods, like RWX mmap-ings. Code is compiled “on-the-fly” by JIT compilers which typically create such mappings as an area to which they first write and then execute. However, this is dangerous because it can open up pathways by which arbitrary code can be injected into a running process. So, PaX does not allow RWX mmap-ings — it doesn’t allow it unless that kernel is told otherwise. This is where the PaX flags come in. In the JIT example, marking the executables with `paxctl-ng -m` will turn off PaX’s MPROTECT and allow the RWX mmap-ing. The issue of consistent PaX markings between executable and their libraries arises when it is the library that needs the markings. But when loaded, it is the markings of the executable, not the library, which set the PaX restrictions on the running process. [9]  So if its the library needs the markings, you have to migrate the markings from the library to the executable. Aha! Here we go again: we need to answer the question “what are all the consumers of a particular library so we can migrate its flags to them?” We can, as revdep-rebuild does, re-read all the ELF objects on the system, reconstruct the directed linkage graph, then invert it; or we can just start from the already gathered VDB information and save some time. Like revdep-rebuild and revdep-rebuild.py, I wrote two utilities. The original, revdep-pax, did forward and reverse migration of PaX flags by gathering information with ldd. It was horribly slow, 5 to 10 minutes depending on the number of objects in $PATH and shared object reported by `ldconfig -p`. I then rewrote it to use VDB information and it accomplished the same task in a fraction of the time [10]. Since constructing and inverting the directed linkage graph is such a useful operation, I figured I’d abstract the bare essential code into a python class which you can get at [11]. The data structure containing the entire graph is a compound python dictionary of the form

{
        abi1 : { path_to_elf1 : [ soname1, soname2, ... ], ... },
        abi2 : { path_to_elf2 : [ soname3, soname4, ... ], ... },
        ...
}

whereas the inverted graph has form

{
        abi1 : { soname1 : [ path_to_elf1, path_to_elf2, ... ], ... },
        abi2 : { soname2 : [ path_to_elf3, path_to_elf4, ... ], ... },
        ...
}

Simple!

Okay, up to now I concentrated on exporting NEEDED.ELF.2 information. So what about rest of the VDB information? Is it useful? A lot of questions regarding Gentoo packages can be answered by “grepping the tree.” If you use portage as your PMS, then the same sort of grep-sed-awk foo magic can be performed on /var/db/pkg to answer similar questions. However, this assumes that the PMS’s cached information is in plain ASCII format. If a PMS decides to use something like Berkeley DB or sqlite, then we’re going to need a tool to read the db format which the PMS itself should provide. Because I do a lot of release engineering of uclibc and musl stages, one need that often comes up is the need to compare of what’s installed in the stage3 tarballs for the various arches and alternative libc’s. So, I run some variation of the following script

#!/usr/bin/env python

import portage, re

portdb = portage.db[portage.root]["vartree"].dbapi

arm_stable = open('arm-stable.txt', 'w')
arm_testing = open('arm-testing.txt', 'w')

for pkg in portdb.cpv_all():
keywords = portdb.aux_get(pkg, ["KEYWORDS"])[0]

arches = re.split('\s+', keywords)
        for a in arches:
                if re.match('^arm$', a):
                        arm_stable.write("%s\n" % pkg)
                if re.match('^~arm$', a):
                        arm_testing.write("%s\n" % pkg)

arm_stable.close()
arm_testing.close()

in a stage3-amd64-uclibc-hardened chroot to see what stable packages in the amd64 tarball are ~arm. [12]  I run similar scripts in other chroots to do pairwise comparisons. This gives me some clue as to what may be falling behind in which arches — to keep some consistency between my various stage3 tarballs. Of course there are other utilities to do the same, like eix, gentoolkit etc, but then one still has to resort to parsing the output of those utilities to get the answers you want. An API for VDB information allows you to write your own custom utility to answer the precise questions you need answers. I’m sure you can multiply these examples.

Let me close with a confession. The above is propaganda for the upcoming GLEP 64 which I just wrote [13]. The purpose of the GLEP is to delineate what information should be exported by all PMS’s with particular emphasis on NEEDED.ELF.2 for the reasons stated above.  Currently portage does provide NEEDED.ELF.2 but paludis does not.  I’m not sure what future PMS’s might or might not provide, so let’s set a standard now for an important feature.

 

Notes:

[1] You can see where NEEDED.ELF.2 is generated for details. Take a look at line ~520 of /usr/lib/portage/bin/misc-functions.sh, or search for the comment “Create NEEDED.ELF.2 regardless of RESTRICT=binchecks”.

[2] A simple hands on tutorial can be found at http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html. It also includes dynamic linking via dlopen() which complicates the nice neat graph that can be constructed from NEEDED.ELF.2.

[3] I’m using the term “directed graph” as defined in graph theory. See http://en.wikipedia.org/wiki/Directed_graph. The nodes of the graph are each ELF object and the directed edges are from the consumer of the shared object to the shared object.

[4] Well, not quite. If you run readelf -d on readelf -d /lib/libc.so.6 you’ll see that it links back to /lib/ld-linux-x86-64.so.2 which doesn’t NEED anything else. The former is stricly your standard C library (man 7 libc) while the later is the dynamic linker/loader (man 8 ld.so).

[5] I should mention parenthatically that there are other executable/library file formats such as Mach-O used on MacOS X. The above arguments translate over to any executable formats which permit shared libraries and dynamic linking. My prejudice for ELF is because it is the primary executable format used on Linux and BSD systems.

[6] I’m coining this term here. If you read the revdep-rebuild code, you won’t see reference to any graph there. Bash doesn’t readily lend itself to the neat data structures that python does.

[7] Just a word of caution, revdep-rebuild.py is still in development and does warn when you run it “This is a development version, so it may not work correctly. The original revdep-rebuild script is installed as revdep-rebuild.sh”.

[8] See https://wiki.gentoo.org/wiki/Hardened/PaX_Quickstart for an explanation of what PaX does as well as how it works.

[9] grep the contents of fs/binfmt_elf.c for PT_PAX_FLAGS and CONFIG_PAX_XATTR_PAX_FLAGS to see how these markings are used when the process is loaded from the ELF object. You can see the PaX protection on a running process by using `cat /proc/<pid>/maps | grep ^PaX` or `pspax` form the pax-utils package.

[10] The latest version off the git repo is at http://git.overlays.gentoo.org/gitweb/?p=proj/elfix.git;a=blob;f=scripts/revdep-pax.

[11] http://git.overlays.gentoo.org/gitweb/?p=proj/elfix.git;a=blob;f=pocs/link-graph/link_graph.py.

[12] These stages are distributed at http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64-uclibc-hardened/ and http://distfiles.gentoo.org/experimental/arm/uclibc/.

[13] https://bugs.gentoo.org/show_bug.cgi?id=518630

Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)

In a previous post, we've already looked at the structure of Perl ebuilds in Gentoo Linux. Now, let's see what happens in the case of a major Perl update.

Does this look familiar?

UPDATE THE PERL MODULES:
After updating dev-lang/perl you must reinstall
the installed perl modules.
Use: perl-cleaner --all
Then maybe you have updated your major Perl version recently, since this important message is printed by emerge afterwards. So, what is it about? In short, a certain disconnect between the "Perl way" of doing things and the rest of the world. Both have their merits, they just don't play very well with each other... and the result is that major Perl updates in Gentoo have traditionally also been a major pain. (This will become much better in the future, see below.)

Let's see where a perl package stores its files.
caipi ~ # equery files dev-perl/Email-Address
 * Searching for Email-Address in dev-perl ...
 * Contents of dev-perl/Email-Address-1.898.0:
/usr
/usr/lib
/usr/lib/perl5
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/vendor_perl/5.16.3
/usr/lib/perl5/vendor_perl/5.16.3/Email
/usr/lib/perl5/vendor_perl/5.16.3/Email/Address.pm
/usr/share
/usr/share/doc
/usr/share/doc/Email-Address-1.898.0
/usr/share/doc/Email-Address-1.898.0/Changes.bz2
/usr/share/doc/Email-Address-1.898.0/README.bz2
caipi ~ #
Interesting- the installation path contains the Perl version! The reasons for upstream to do this are pretty much obvious, the application binary interface for compiled modules can change and it's necessary to keep the installed modules for different versions apart. Also, in theory you can keep different Perl versions installed in parallel. Nice idea, however if you have only one "system Perl" installation, and you exchange that for a newer version (say, 5.18.1 instead of 5.16.3), the result is that the new version won't find the installed packages anymore.

The results are rather annoying. Imagine you haven't updated your system for a while, one of the many packages to be updated is dev-lang/perl, and later maybe (just picking an example at random) gnome-base/gsettings-desktop-schemas. Perl is updated fine, but when portage arrives at building the gnome package, the build fails with something like
checking for perl >= 5.8.1... 5.18.2
checking for XML::Parser... configure: error: XML::Parser perl module is required for intltool
Right. Perl is updated, dev-perl/XML-Parser is still installed in the old path, and Perl doesn't find it. Bah.

Enter perl-cleaner, the traditional "solution". This small program checks for files in "outdated" Perl installation paths, finds out which packages they belong to, and makes portage rebuild the corresponding packages. During the rebuild, the installation is run by the updated Perl, which makes the files go into the new, now correct path.

This sounds like a good solution, but there are a lot of details and potential problems hidden. For once, most likely you'll run perl-cleaner after a failed emerge command, and some unrelated packages still need updates. Portage will try to figure out how to do this, but blockers and general weirdness may happen. Then, sometimes a package isn't needed with the new Perl version anymore, but perl-cleaner can't know that. Again the result may be a blocker. We've added the following instructions to the perl-cleaner output, which may help cleaning up the most frequent difficulties:
 * perl-cleaner is stopping here:
 * Fix the problem and start perl-cleaner again.
 *
 * If you encounter blockers involving virtuals and perl-core, here are
 * some things to try:
 *   Remove all perl-core packages from your world file
 *     emerge --deselect --ask $(qlist -IC 'perl-core/*')
 *   Update all the installed Perl virtuals
 *     emerge -uD1a $(qlist -IC 'virtual/perl-*')
 *   Afterwards re-run perl-cleaner
In the end, you may have to try several repeated emerge and perl-cleaner commands until you have an updated and consistent system again. So far, it always worked somehow with fiddling, but the situation was definitely not nice.

So what's the future? Well...

EAPI=5 brings the beautiful new feature of subslots and slot operator dependencies. In short, a package A may declare a subslot, and a package B that depends on A may declare "rebuild me if A changes subslot". This mechanism is now used to automate the Perl rebuilds directly from within emerge: dev-lang/perl declares a subslot corresponding to its major version, say "5.18", and every package that installs Perl modules needs to depend on it with the subslot-rebuild requested, e.g.
RDEPEND="dev-lang/perl:="
The good news about this is that portage now knows the dependency tree and can figure out the correct reinstallation order.

The bad news is, it can only work perfectly after all Perl packages have been converted to EAPI=5 and stabilized. perl-core is done, but with about 2100 ebuilds that use perl-module.eclass in the portage tree still quite some work remains. I've plotted the current EAPI distribution of ebuilds using perl-module.eclass in a pie chart for illustration... Maybe we're done when Perl 5.20 goes stable. Who knows. :)

August 01, 2014
Sven Vermeulen a.k.a. swift (homepage, bugs)
Gentoo Hardened July meeting (August 01, 2014, 19:48 UTC)

I failed to show up myself (I fell asleep – kids are fun, but deplete your energy source quickly), but that shouldn’t prevent me from making a nice write-up of the meeting.

Toolchain

GCC 4.9 gives some issues with kernel compilations and other components. Lately, breakage has been reported with GCC 4.9.1 compiling MySQL or with debugging symbols. So for hardened, we’ll wait this one out until the bugs are fixed.

For GCC 4.10, the –enable-default-pie patch has been sent upstream. If that is accepted, the SSP one will be sent as well.

In uclibc land, stages are being developed for PPC. This is the final architecture that is often used in embedded worlds that needed support for it in Gentoo, and that’s now being finalized. Go blueness!

SELinux

A libpcre upgrade broke relabeling operations on SELinux enabled systems. A fix for this has been made part of libselinux, but a little too late, so some users will be affected by the problem. It’s easily worked around (removing the *.bin files in the contexts/files/ directory of the SELinux configuration) and hopefully will never occur again.

The 2.3 userland has finally been stabilized (we had a few dependencies that we were waiting for – and we were a dependency ourselves for other packages as well).

Finally, some thought discussion is being done (not that there’s much feedback on it, but every documented step is a good step imo) on the SELinux policy within Gentoo (and the principles that we’ll follow that are behind it).

Kernel and grsecurity / PaX

Due to some security issues, the Linux kernel sources have been stabilized more rapidly than usual, which left little time for broad validation and regression testing. Updates and fixes have been applied since and new stabilizations occurred. Hopefully we’re now at the right, stable set again.

The C-based install-xattr application (which is performance-wise a big improvement over the Python-based one) is working well in “lab environments” (some developers are using it exclusively). It is included in the Portage repository (if I understand the chat excerpts correctly) but as such not available for broader usage yet.

An update against elfix is made as well as there was a dependency mismatch when building with USE=-ptpax. This will be corrected in elfix-0.9.

Finally, blueness is also working on a GLEP (Gentoo Linux Enhancement Proposal) to export VDB information (especially NEEDED.ELF.2) as this is important for ELF/library graph information (as used by revdep-pax, migrate-pax, etc.). Although Portage already does this, this is not part of the PMS and as such other package managers might not do this (such as Paludis).

Profiles

Updates on the profiles has been made to properly include multilib related variables and other metadata. For some profiles, this went as easy as expected (nice stacking), but other profiles have inheritance troubles making it much harder to include the necessary information. Although some talks have arised on the gentoo-dev mailinglist about refactoring how Gentoo handles profiles, there hasn’t been done much more than just talking :-( But I’m sure we haven’t heard the last of this yet.

Documentation

Blueness has added information on EMULTRAMP in the kernel configuration, especially noting to the user that it is needed for Python support in Gentoo Hardened. It is also in the PaX Quickstart document, although this document is becoming a very large one and users might overlook it.

Alexys Jacob a.k.a. ultrabug (homepage, bugs)
Europython 2014 (August 01, 2014, 14:29 UTC)

I had the chance to participate to europython 2014 as my company was sponsoring the event.

IMG_20140725_161445-1024x576

This was a great week where I got to meet some very interesting people and hear about some neat python use cases, libraries and new technologies so I thought I’d write a quick summary of my biased point of view.

ZeroMQ

I had the chance to meet Pieter Hintjens and participate in a 3 hours workshop on ZeroMQ. This was very interesting and refreshing as to go in more depth into this technology which I’ve been using in production for several years now.

Pieter is also quite a philosophical person and I strongly encourage you to listen to his keynote. I ended up pinging him in real life for an issue I’ve been waiting for bug correction on the libzmq and it got answered nicely.

uWSGI

Another big thing in our python stack is the uWSGI application container which I love and follow closely even if my lack of knowledge in C++ prevents me for going too deep in the source code… I got the chance to speak with Roberto De Ioris about the next 2.1 release and propose him two new features.

Thanks a lot for your consideration Roberto !

Trends

  • Not tested = broken !
  • Python is strong and very lively in the Big Data world
  • Asynchronous and distributed architectures get more and more traction and interest

Videos

All the talks videos are already online, you should check them out !

July 27, 2014
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)


We've got the stabilization of Perl 5.18 upcoming, so what better chance is there to explain a bit how the Perl-related ebuilds in Gentoo work...

First of all, there is dev-lang/perl. This contains the Perl core distribution, installing the binaries and all the Perl modules that are bundled with Perl itself.

Then, there is the perl-core category. It contains independent ebuilds for Perl modules that are also present in the core Perl distribution. Most Perl modules that are bundled with Perl are also in addition released as independent tarballs. If any of these packages is installed from perl-core, its files are placed such that the perl-core download overrides the bundled copy. This means you can also update part of the bundled Perl modules, e.g. in case of a bug, without updating Perl itself.

Next, there are a lot of virtuals "virtual/perl-..." in the virtual category of the portage tree. What are these good for? Well, imagine you want to depend on a specific version of a module that is usually bundled with Perl. For example, you need Module::CoreList at at least version 3.  This can either be provided by a new enough Perl (for example, now hardmasked Perl 5.20 contains Module::CoreList 3.10), or by a separate package from perl-core (where we have Module::CoreList 5.021001 as perl-core/Module-CoreList-5.21.1).
To make sure that everything works, you should never directly depend on a perl-core package, but always on the corresponding virtual (here virtual/perl-Module-CoreList; any perl-core package must have a corresponding virtual). Then both ways to fulfil the dependency are automatically taken into account. (Future repoman versions will warn if you directly depend on perl-core. Also you should never have anything perl-core in your world file!)

Last, we have lots of lots of modules in the dev-perl category. Most of them are from CPAN, and the only thing they have in common is that they have no copy inside core Perl.

I hope this clarifies things a bit. More Perl posts coming...

July 22, 2014
Arun Raghavan a.k.a. ford_prefect (homepage, bugs)

One of the first tools that you should get if you’re hacking with GStreamer or want to play with the latest version without doing evil things to your system is probably the gst-uninstalled script. It’s the equivalent of Python’s virtualenv for hacking on GStreamer. :)

The documentation around getting this set up is a bit frugal, though, so here’s my attempt to clarify things. I was going to put this on our wiki, but that’s a bit search-engine unfriendly, so probably easiest to just keep it here. The setup I outline below can probably be automated further, and comments/suggestions are welcome.

  • First, get build dependencies for GStreamer core and plugins on your distribution. Commands to do this on some popular distributions follow. This will install a lot of packages, but should mean that you won’t have to play find-the-plugin-dependency for your local build.

    • Fedora: $ sudo yum-builddep gstreamer1-*
    • Debian/Ubuntu: $ sudo apt-get build-dep gstreamer1.0-plugins-{base,good,bad,ugly}
    • Gentoo: having the GStreamer core and plugin packages should suffice
    • Others: drop me a note with the command for your favourite distro, and I’ll add it here
  • Next, check out the code (by default, it will turn up in ~/gst/master)

    • $ curl http://cgit.freedesktop.org/gstreamer/gstreamer/plain/scripts/create-uninstalled-setup.sh | sh
    • Ignore the pointers to documentation that you see — they’re currently defunct
  • Now put the gst-uninstalled script somewhere you can get to it easily:

    • $ ln -sf ~/gst/master/gstreamer/scripts/gst-uninstalled ~/bin/gst-master
    • (the -master suffix for the script is important to how the script works)
  • Enter the uninstalled environment:

    • $ ~/bin/gst-master
    • (this puts you in the directory with all the checkouts, and sets up a bunch of environment variables to use your uninstalled setup – check with echo $GST_PLUGIN_PATH)
  • Time to build

    • $ ./gstreamer/scripts/git-update.sh
  • Take it out for a spin

    • $ gst-inspect-1.0 filesrc
    • $ gst-launch-1.0 playbin uri=file:///path/to/some/file
    • $ gst-discoverer-1.0 /path/to/some/file
  • That’s it! Some tips:

    • Remember that you need to run ~/bin/gst-master to enter the environment for each new shell
    • If you start up a GStreamer app from your system in this environment, it will use your uninstalled libraries and plugins
    • You can and should periodically update you tree by rerunning the git-update.sh script
    • To run gdb on gst-launch, you need to do something like:
    • $ libtool --mode=execute gdb --args gstreamer/tools/gst-launch-1.0 videotestsrc ! videoconvert ! xvimagesink
    • I find it useful to run cscope on the top-level tree, and use that for quick code browsing

July 19, 2014
Paweł Hajdan, Jr. a.k.a. phajdan.jr (homepage, bugs)

I was experimenting in my arm chroot, and after a gcc upgrade and emerge --depclean --ask that removed the old gcc I got the following error:

# ls -l
ls: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory

Fortunately the newer working gcc was present, so the steps to make things work again were:

# LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.8.2/" gcc-config -l
 * gcc-config: Active gcc profile is invalid!

 [1] armv7a-hardfloat-linux-gnueabi-4.8.2

# LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.8.2/" gcc-config 1 
 * Switching native-compiler to armv7a-hardfloat-linux-gnueabi-4.8.2 ...

Actually my first thought was using busybox. The unexpected breakage during a routine gcc upgrade made me do some research in case I can't rely on /bin/busybox being present and working.

I highly recommend the following links for further reading:
http://lambdaops.com/rm-rf-remains
http://eusebeia.dyndns.org/bashcp
http://www.reddit.com/r/linux/comments/27is0x/rm_rf_remains/ci199bk

Read more »

July 14, 2014
Sebastian Pipping a.k.a. sping (homepage, bugs)

I just watched a TED talk that I would like to share with you.

First, let me quote a line from that talk that works without much context:

Next to the technologie, entertainment and social media industries
we now spend more time with other people’s ideas than we do with our own.

For the remainder, see for yourself: Blur the line: Dan Jaspersen at TEDxCSU

Richard Freeman a.k.a. rich0 (homepage, bugs)
Quick systemd-nspawn guide (July 14, 2014, 20:31 UTC)

I switched to using systemd-nspawn in place of chroot and wanted to give a quick guide to using it.  The short version is that I’d strongly recommend that anybody running systemd that uses chroot switch over – there really are no downsides as long as your kernel is properly configured.

Chroot should be no stranger to anybody who works on distros, and I suspect that the majority of Gentoo users have need for it from time to time.

The Challenges of chroot

For most interactive uses it isn’t sufficient to just run chroot.  Usually you need to mount /proc, /sys, and bind mount /dev so that you don’t have issues like missing ptys, etc.  If you use tmpfs you might also want to mount the new tmp, var/tmp as tmpfs.  Then you might want to make other bind mounts into the chroot.  None of this is particularly difficult, but you usually end up writing a small script to manage it.

Now, I routinely do full backups, and usually that involves excluding stuff like tmp dirs, and anything resembling a bind mount.  When I set up a new chroot that means updating my backup config, which I usually forget to do since most of the time the chroot mounts aren’t running anyway.  Then when I do leave it mounted overnight I end up with backups consuming lots of extra space (bind mounts of large trees).

Finally, systemd now by default handles bind mounts a little differently when they contain other mount points (such as when using –rbind).  Apparently unmounting something in the bind mount will cause systemd to unmount the corresponding directory on the other side of the bind.  Imagine my surprise when I unmounted my chroot bind to /dev and discovered /dev/pts and /dev/shm no longer mounted on the host.  It looks like there are ways to change that, but this isn’t the point of my post (it just spurred me to find another way).

Systemd-nspawn’s Advantages

Systemd-nspawn is a tool that launches a container, and it can operate just like chroot in its simplest form.  By default it automatically sets up most of the overhead like /dev, /tmp, etc.  With a few options it can also set up other bind mounts as well.  When the container exits all the mounts are cleaned up.

From the outside of the container nothing appears different when the container is running.  In fact, you could spawn 5 different systemd-nspawn container instances from the same chroot and they wouldn’t have any interaction except via the filesystem (and that excludes /dev, /tmp, and so on – only changes in /usr, /etc will propagate across).  Your backup won’t see the bind mounts, or tmpfs, or anything else mounted within the container.

The container also has all those other nifty container benefits like containment – a killall inside the container won’t touch anything outside, and so on.  The security isn’t airtight – the intent is to prevent accidental mistakes.  

Then, if you use a compatible sysvinit (which includes systemd, and I think recent versions of openrc), you can actually boot the container, which drops you to a getty inside.  That means you can use fstab to do additional mounts inside the container, run daemons, and so on.  You get almost all the benefits of virtualization for the cost of a chroot (no need to build a kernel, and so on).  It is a bit odd to be running systemctl poweroff inside what looks just like a chroot, but it works.

Note that unless you do a bit more setup you will share the same network interface with the host, so no running sshd on the container if you have it on the host, etc.  I won’t get into this but it shouldn’t be hard to run a separate network namespace and bind the interfaces so that the new instance can run dhcp.

How to do it

So, getting it actually working will likely be the shortest bit in this post.

You need support for namespaces and multiple devpts instances in your kernel:

CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y

 From there launching a namespace just like a chroot is really simple:

systemd-nspawn -D .

That’s it – you can exit from it just like a chroot.  From inside you can run mount and see that it has taken care of /dev and /tmp for you.  The “.” is the path to the chroot, which I assume is the current directory.  With nothing further it runs bash inside.

If you want to add some bind mounts it is easy:

systemd-nspawn -D . –bind /usr/portage

Now your /usr/portage is bound to your host, so no need to sync/etc.  If you want to bind to a different destination add a “:dest” after the source, relative to the root of the chroot (so –bind foo is the same as –bind foo:foo).

If the container has a functional init that can handle being run inside, you can add a -b to boot it:

systemd-nspawn -D . –bind /usr/portage -b

Watch the init do its job.  Shut down the container to exit.

Now, if that container is running systemd you can direct its journal to the host journal with -h:

systemd-nspawn -D . –bind /usr/portage -j -b

Now, nspawn registers the container so that it shows up in machinectl.  That makes it easy to launch a new getty on it, or ssh to it (if it is running ssh – see my note above about network namespaces), or power it off from the host.  

That’s it.  If you’re running systemd I’d suggest ditching chroot almost entirely in favor of nspawn.  


Filed under: foss, gentoo, linux

Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
Biggest ebuilds in-tree (July 14, 2014, 06:39 UTC)

Random datapoint: There's only about 10 packages with ebuilds over 600 lines.

Sorted by lines, duplicate entries per-package removed, these are the biggest ones:

828 dev-lang/ghc/ghc-7.6.3-r1.ebuild
817 dev-lang/php/php-5.3.28-r3.ebuild
750 net-nds/openldap/openldap-2.4.38-r2.ebuild
664 www-client/chromium/chromium-36.0.1985.67.ebuild
654 www-servers/nginx/nginx-1.4.7.ebuild
658 games-rpg/nwn-data/nwn-data-1.29-r5.ebuild
654 media-video/mplayer/mplayer-1.1.1-r1.ebuild
644 dev-vcs/git/git-9999-r3.ebuild
621 x11-drivers/ati-drivers/ati-drivers-13.4.ebuild
617 sys-freebsd/freebsd-lib/freebsd-lib-9.1-r11.ebuild

July 13, 2014
Sebastian Pipping a.k.a. sping (homepage, bugs)

Background story / context

At work I’m dealing with a test suite running >30 minutes, even on moderately fast hardware. When testing some changes, I launch the test suite and start working on something else to not be waiting for the test suite. Now the sooner I know that the test suite finished execution, the sooner I can fix errors and give it another spin. So checking the test suite for being done manually is not efficient.

The problem

What I wanted was a notification, something audible, looped, like an alarm clock. Either

$ ALARM_WHEN_DONE cmd [p1 p2 ..]

or

$ cmd [p1 p2 ..] ; ALARM

usage would have worked for me.

My approach

I ended up grabbing the free Analog Alarm Clock sound — the low-quality MP3 version download works without registration — and this shell alias:

alias ALARM='mplayer --loop=0 ~/Desktop/alarm.mp3 &>/dev/null'

With this alias, now I can do stuff like

./testrunner ; ALARM

on the shell and never miss the end of test suite execution again :)

Do you have a different/better approach to the same problem? Let me know!

PS: Yes, I have heard of continuous integration and we do that, too :)

July 12, 2014
Hanno Böck a.k.a. hanno (homepage, bugs)
LibreSSL on Gentoo (July 12, 2014, 18:31 UTC)

LibreSSL PuffyYesterday the LibreSSL project released the first portable version that works on Linux. LibreSSL is a fork of OpenSSL and was created by the OpenBSD team in the aftermath of the Heartbleed bug.

Yesterday and today I played around with it on Gentoo Linux. I was able to replace my system's OpenSSL completely with LibreSSL and with few exceptions was able to successfully rebuild all packages using OpenSSL.

After getting this running on my own system I installed it on a test server. The Webpage tlsfun.de runs on that server. The functionality changes are limited, the only thing visible from the outside is the support for the experimental, not yet standardized ChaCha20-Poly1305 cipher suites, which is a nice thing.

A warning ahead: This is experimental, in no way stable or supported and if you try any of this you do it at your own risk. Please report any bugs you have with my overlay to me or leave a comment and don't disturb anyone else (from Gentoo or LibreSSL) with it. If you want to try it, you can get a portage overlay in a subversion repository. You can check it out with this command:
svn co https://svn.hboeck.de/libressl-overlay/
git clone https://github.com/gentoo/libressl.git

This is what I had to do to get things running:

LibreSSL itself

First of all the Gentoo tree contains a lot of packages that directly depend on openssl, so I couldn't just replace that. The correct solution to handle such issues would be to create a virtual package and change all packages depending directly on openssl to depend on the virtual. This is already discussed in the appropriate Gentoo bug, but this would mean patching hundreds of packages so I skipped it and worked around it by leaving a fake openssl package in place that itself depends on libressl.

LibreSSL deprecates some APIs from OpenSSL. The first thing that stopped me was that various programs use the functions RAND_egd() and RAND_egd_bytes(). I didn't know until yesterday what egd is. It stands for Entropy Gathering Daemon and is a tool written in perl meant to replace the functionality of /dev/(u)random on non-Linux-systems. The LibreSSL-developers consider it insecure and after having read what it is I have to agree. However, the removal of those functions causes many packages not to build, upon them wget, python and ruby. My workaround was to add two dummy functions that just return -1, which is the error code if the Entropy Gathering Daemon is not available. So the API still behaves like expected. I also posted the patch upstream, but the LibreSSL devs don't like it. So on the long term it's probably better to fix applications to stop trying to use egd, but for now these dummy functions make it easier for me to build my system.

The second issue popping up was that the libcrypto.so from libressl contains an undefined main() function symbol which causes linking problems with a couple of applications (subversion, xorg-server, hexchat). According to upstream this undefined symbol is intended and most likely these are bugs in the applications having linking problems. However, for now it was easier for me to patch the symbol out instead of fixing all the apps. Like the egd issue on the long term fixing the applications is better.

The third issue was that LibreSSL doesn't ship pkg-config (.pc) files, some apps use them to get the correct compilation flags. I grabbed the ones from openssl and adjusted them accordingly.

OpenSSH

This was the most interesting issue from all of them.

To understand this you have to understand how both LibreSSL and OpenSSH are developed. They are both from OpenBSD and they use some functions that are only available there. To allow them to be built on other systems they release portable versions which ship the missing OpenBSD-only-functions. One of them is arc4random().

Both LibreSSL and OpenSSH ship their compatibility version of arc4random(). The one from OpenSSH calls RAND_bytes(), which is a function from OpenSSL. The RAND_bytes() function from LibreSSL however calls arc4random(). Due to the linking order OpenSSH uses its own arc4random(). So what we have here is a nice recursion. arc4random() and RAND_bytes() try to call each other. The result is a segfault.

I fixed it by using the LibreSSL arc4random.c file for OpenSSH. I had to copy another function called arc4random_stir() from OpenSSH's arc4random.c and the header file thread_private.h. Surprisingly, this seems to work flawlessly.

Net-SSLeay

This package contains the perl bindings for openssl. The problem is a check for the openssl version string that expected the name OpenSSL and a version number with three numbers and a letter (like 1.0.1h). LibreSSL prints the version 2.0. I just hardcoded the OpenSSL version numer, which is not a real fix, but it works for now.

SpamAssassin

SpamAssassin's code for spamc requires SSLv2 functions to be available. SSLv2 is heavily insecure and should not be used at all and therefore the LibreSSL devs have removed all SSLv2 function calls. Luckily, Debian had a patch to remove SSLv2 that I could use.

libesmtp / gwenhywfar

Some DES-related functions (DES is the old Data Encryption Standard) in OpenSSL are available in two forms: With uppercase DES_ and with lowercase des_. I can only guess that the des_ variants are for backwards compatibliity with some very old versions of OpenSSL. According to the docs the DES_ variants should be used. LibreSSL has removed the des_ variants.

For gwenhywfar I wrote a small patch and sent it upstream. For libesmtp all the code was in ntlm. After reading that ntlm is an ancient, proprietary Microsoft authentication protocol I decided that I don't need that anyway so I just added --disable-ntlm to the ebuild.

Dovecot

In Dovecot two issues popped up. LibreSSL removed the SSL Compression functionality (which is good, because since the CRIME attack we know it's not secure). Dovecot's configure script checks for it, but the check doesn't work. It checks for a function that LibreSSL keeps as a stub. For now I just disabled the check in the configure script. The solution is probably to remove all remaining stub functions. The configure script could probably also be changed to work in any case.

The second issue was that the Dovecot code has some #ifdef clauses that check the openssl version number for the ECDH auto functionality that has been added in OpenSSL 1.0.2 beta versions. As the LibreSSL version number 2.0 is higher than 1.0.2 it thinks it is newer and tries to enable it, but the code is not present in LibreSSL. I changed the #ifdefs to check for the actual functionality by checking a constant defined by the ECDH auto code.

Apache httpd

The Apache http compilation complained about a missing ENGINE_CTRL_CHIL_SET_FORKCHECK. I have no idea what it does, but I found a patch to fix the issue, so I didn't investigate it further.

Further reading:
Someone else tried to get things running on Sabotage Linux.

Update: I've abandoned my own libressl overlay, a LibreSSL overlay by various Gentoo developers is now maintained at GitHub.