<UU> Somedays, I think why can't we have computers which just work.
<UU> But then I remember that I am a Computer Scientist.
<UU> So, yeah, I guess I understand why.
<Nirbheek> :D

Gentoo Prefix is still alive and going strong. In my opinion, Gentoo Prefix remains a strong point of Gentoo Linux and really establishes that Gentoo Linux is a metadistribution. In this post I want to focus on the numbers. The number of packages in the Gentoo Prefix tree, specifically. But first, a history lesson. It wasn’t until EAPI3 in Gentoo that “allowed” Gentoo Prefix variables into the main Gentoo Linux tree. That was in late 2011, but Gentoo Prefix existed much before then, all the way back to 2006 (at least). Before EAPI3, the prefix team made slight modifications to ebuilds and placed them in a repo and called it the tree of packages for Gentoo Prefix. This worked fine, but we had growing pains. The major issue was that we were getting too successful to manage the increased contributions from users. In other words, as the number of “forked” packages grew, the amount of maintenance time increased greatly – this is due to the fact that it is a chore to keep our forks synced. At least, a large chore for a small team. This is why we looked for help and adoption from the other pool of 200 Gentoo Developers, hence EAPI3 and beyond. Since supporting Gentoo Prefix is not a big use of overall developer time, this has gone over quite well in my opinion – yes, there are some pain points at times I do realize. Enough history, here are the numbers:

• Number of packages in Gentoo Linux: 15554 packages in 154 categories.
• Number of total* packages in Gentoo Prefix: 9483 packages in 154 categories.
• Number of KEYWORDED packages in Gentoo Prefix: About 3000 for the most popular arch
• Number of packages still NOT in the main Gentoo Linux tree: 369 packages

* The total packages in the tree also contains non-keyworded packages because that just makes life simple. Once packages started migrating to the main tree, I helped think of this “whitelist” concept. The short version of the whitelist is that if a package is listed in that text file, it gets included in the Gentoo Prefix tree as a direct copy of the version in the Gentoo Linux tree. The presense of the package in the old repo means that it is used instead. Eventually, this concept will go away and we will overlay the Gentoo Linux tree directly.

So why is it taking so long to migrate ALL packages to the Gentoo Linux tree? Well, that is where the rubber meets the road and we get into roadblocks. A roadblock for us could be a number of things, such as a disagreement with the Gentoo Linux maintainer, some patches existing that we don’t feel are a good fit for Gentoo Linux, or even us being lazy and not submitting stuff to upstream. We also don’t want to push invasive changes to Gentoo Linux for critical packages, like the toolchain for example.

It has long since been our agenda to not add anymore packages to the old repo and going forward only adding new stuff to Gentoo Linux directly. I hope we can make a dent in those remaining 369 in 2012!

If you do, maybe you want to consider proxy-maintaining it as it now on its way out. Upstream has a much newer version available, and we in the Proxy Maintainers team will be glad to steer you in the right direction when you need the help.

Just send us an email.

# Andreas K. Hüttel <dilfridge@gentoo.org> (27 Jan 2012)
# Has developed into an unmaintainable mess, and everyone who
# knows about it is either retired or missing in action.
# Several minor bugs and one ugly security issues (#386271).
# Masked for removal because of lack of maintainer.
# Please try app-text/epdfview as light-weight replacement.
app-text/xpdf

• app-text/zathura (based on poppler & gtk+)
• app-text/apvlv (based on poppler & gtk+)
• app-text/epdfview (based on poppler & gtk+, ~unmaintained)
• app-text/mupdf
• app-text/gv (based on ghostscript)
• app-text/evince (based on poppler, Gnome application)
• kde-base/okular (based on poppler, KDE application)
• app-text/acroread (yes I know...)

I started looking at ModSecurity when I wanted to implement a Uesr-Agent based antispam method which has proven time and time again working quite well to the point I started publishing the ruleset which takes care not only of working as an antispam method, as well as a way to avoid tons of bad crawlers from finding my email addresses and so on.

When I first proposed this kind of filtering I received quite a few complains, that the HTTP protocol didn’t define the User-Agent in such a way, but thanks first to EFF’s Panopticlick – demonstrating clearly that the “anonymised” requests are not as anonymous as their perpetrators would expect them to be – and most recently SpiderLabs’s work I am now fully certain that I took the right road.

I’ve spent a bit more work on the rules this week, to make them further resilient to fake the requests such as those coming from scriptkiddies’ tools such as the HOIC tool described in the SpiderLabs’s blog post linked above. One of the most interesting detection I came up with is for real Chrome requests: while it seems to me like Google itself does not leverage it, Chrome as of version 18 is still implementing their own proposed Shared Dictionary Compression for HTTP even though I don’t think it’ll ever be used in the real world. Being the only browser actually requesting such an encoding, I can easily assume a connection between the two — this was only disattended by Epiphany, which in its most recent versions declares to be Chrome… which means you then have a browser claiming to be another (Chrome), which in turn claims to be a third (Safari), which uses an engine (KHTML) claiming to be the same as another (Gecko), all the while declaring it’s all compatible with Mozilla/5.0.

One issue I found while doing this work had to do with Android. For both versions 2 and 3 (is somebody really hoping to use Android 1?), the (default, AOSP) browser sends a full-fledged HTTP request, which among other things include an Accept header. This is what every browser I ever tried does, to the point that ModSecurity’s own Core Rule Set assigns negative points to requests coming without one; in my ruleset it’s further tightened by checking whether the request is purportedly from a known browser, and if so rejecting it if it doesn’t include that header; this worked up to now — note that requests coming through a Proxy, making that explicit through a Via header, are not validated against these checks simply because many proxies are known to muck with the headers.

Anyway as I was saying this is disattended badly by Android 4 (up to 4.0.3, and CyanogenMod as well); it might have started as a way to minimise the bandwidth usage, but for whatever reason in this version, the AOSP browser does not send an Accept reader at all — actually it seems like it dropped most of the headers that it was sending before and that are not strictly necessary for the server to process the request. I could have sworn that Accept was mandatory for the HTTP protocol, but it seems that either I was totally mistaken, or it was only noted in some recommendation that never made it to the standard. The ruleset now exonerates Android 4 from that particular test, but I’m not really too happy about it.

But that’s definitely not the only thing that is out of place with Android. Indeed, if you take an HTC Android device, the browser you open is not the AOSP one, but it’s HTC’s own implementation. This version … does not fully declare itself as an Android device, using a browser compatible with Mobile Safari. Instead, what it reports itself as is a complete Safari, and not in the way that Chrome does it, but by pretending it’s Mac OS X 10.6.3 running on an Intel Mac. Honestly, that’s way crazy to do.

There are a few more things that I hope to be able to handle in my ruleset to make it even tighter, without adding substantial false positives. This means not only fewer spam comments, but also fewer crawlers finding our email addresses, and fewer risks associated with Denial of Service attacks, distributed or not.

If you would like to help with the ruleset, you can find it on Flattr where it’s depressingly stopping at only two clicks. If you would like to use the ruleset, you can find it on GitHub and you can use it for free, obviously.

As discussed in the last Gentoo Qt meeting, we moved our overlay from gitorious to git.overlays.gentoo.org. This is going to be the final move, I promise

Along with that, we decided to change the overlay from qting-edge to just qt. Layman list is alreay updated, so if you still have the old one, you should remove it and add the new one:

# layman -f
# layman -d qting-edge
# layman -a qt

Keep in mind that this overlay contains mostly live ebuilds of Qt (branches 4.7 and master), so make sure that you really need it before blindly adding it (the same applies for the kde overlay). Enjoy!

1. Roll call

johu, hwoarang, pesa, tampakrap, wired

2. Qt 4.8

* cairo fails to build, patched ebuild available in qting-edge, #380013

Cairo build issue is fixed in qting-edge overlay, will be moved together with Qt 4.8.0 to tree.

* qt now defaults to the raster graphicssystem, we should remove raster USE flag, #398283

Wired created a eselect module to choose the Qt graphicsystem. Raster is default, other selectable are opengl, openvg and native. Raster use flag is not needed anymore, qt-gui depends on the new eselect module.

* do we really want to keep qpa USE flag?

qpa and c++0x will be masked in tree.

* are we going to fix #363939 for 4.8?

Wired fixed this bug in qt 4.8.0. Qt 4.8 will be moved to tree on next weekend. Dilfridge prepares kde-base/kstyles-4.7.4 to be rebuild together with Qt 4.8.0 to prevent crashes in KDE apps with Oxygen style.

3. Minor arches and Qt >= 4.7

Upstream supports official amd64, arm and x86, but other arches also considered in configure script. Keep stable keywords for minor arches in Qt 4.6. Wait for minor arches arm, ppc, ppc64 in current stabilization in Qt 4.7.4. Drop sparc keywords in Qt 4.8.0.

4. Overlay migration to git.overlays.gentoo.org

Tampakrap will set up overlay on git.overlays.gentoo.org on next weekend. The new overlay will be renamed to qt instead of qting-edge.

5. Open bugs

* #398885 qdoc3 broken on arm

We will ask the reporter if it works when he builds manually by providing him a configure command to make sure he tries the proper build.

* #394533 Libreoffice crashes in qt on exit

Can’t be reproduced with Libreoffice 3.5.0.1, seems to be resolved by upstream.

* #392433 desktop file name issues

Will be fixed in Qt 4.8.0, so that qt-gui and qt-assistant no longer pass absolute paths to make_desktop_entry().

* #388551 qt-gui[gtkstyle] should depend on gnome-base/libgnomeui-2

We will add a elog message in qt-gui[gtkstyle] saying that for things to work you either need libgnomeui or that variable set properly in your env.

* #382559 qt_mkspecs_dir() returns bad spec directory

The bug will be marked as RESOLVED WORKSFORME, because we can’t reproduce it. Additionally we change the eclass not to use LIBDIR in favor of get_libdir() after Qt 4.8 hits the portage tree.

* #359391 qt4-build.eclass should check for —buildpkgonly before downgrade sanity check

Resolution will be RESOLVED WONTFIX. Sanity check is there for a reason. It’s not a matter of source or binary downgrade.

Since one hour ago, Qt-4.8.0 is in Gentoo portage tree. New major release so lots of new (or broken) stuff. The most important feature in this release is the integration of a new eselect module. This module will allow you to set your default graphics engine without the need to recompile Qt (x11-libs/qt-gui to be precise) from scratch. So, provided you have qt-gui-4.8.0 installed, you should be able to use the eselect module as follows:

hwoarang@mystical ~$ eselect qtgraphicssystem list

Available Qt Graphics Systems:
 [1] native
 [2] opengl
 [3] raster *

(note: if you have x11-libs/qt-openvg installed, one more option should be available)

Simply select your graphics system of preference, and then logout and login again.

hwoarang@mystical ~$ eselect qtgraphicssystem set 2
Setting opengl as your active Qt Graphics System... done
Please logout for changes to take effect.

Thanks to Alex(wired) for the eselect module implementation.
Enjoy ;-)

A small notification to tell you that the SELinux policies that were pushed to the main tree 30 days (or more) ago have now been stabilized (none of them introduced problems, although some of them have other bugs still open which are either fixed in ~arch or will be fixed in the hardened-dev overlay soon). I’ll be working on pushing an additional set of changes to hardened-dev overlay today as it includes fixes for openrc that are quite important, and might even push this to the tree faster than usual.

The reference policy is also working on a new release, so the moment it is released we will be picking that up as well (give or take a month, since my availability will be a bit less the next month).

You might remember that a couple of years ago I ranted about my choice of a Dell laptop — I have not found the time until now to write a full retraction of that post, but you might have guessed that I’m not that bothered by the laptop anymore.

Indeed, after a few rough months, the laptop is working quite nicely nowadays; not only the issues with PME I reported were solved a version of the kernel in or two, but also nowadays gentoo-sources have a (patched) experimental driver for the touchpad that lets me disable it exactly like I wish to. After a firmware upgrade (which is unfortunately only available for Windows, but it’s a small price to pay), both the contactful and the contactless smartcard reader interfaces work fine, the SD card reader works nicely with modern kernel, and so does the soundcard (both speakers and microphones). Even the HSDPA modem (that I bought last year, separately, and was quite easy to set up!) works fine on Linux, even though I haven’t found a way to set up the GPS, or to read/send SMS, not that I care about the latter.

Indeed I haven’t run Windows in there for quite a bit, especially since last time I tried to repartition it I couldn’t get grub2 and Windows 7 to play well together, so I just let it “rotting” for the moment, and I’m now honestly considering whether I want to keep Windows 7 in there – it has a few uses for me at customers’, other than updating BIOS and various devices’ firmware – or just install an SSD and be done with it. Third option would be to find an HDD-in-Optical-Bay adapter and get an SSD for Linux and a (pluggable) HDD for Windows 7.

Anyway, after all this I’m pretty happy with Dell, to the point that I both started suggesting it for my customers, and got a few more things from them (namely a Vostro 3750 laptop to use for Windows development, and an U2711 monitor). Why did I change my mind so completely? Mostly because I have seen how other vendors seem to make it more and more inconvenient to use them for anything but looking at facebook.

Take HP: I had to downgrade a laptop for a customer last week, from Vista to XP. It was not the first time I did that, and not the first time I had to do so to an HP laptop.. but this time it got even worse than usual. Let’s ignore the fact that HP pretends that a ton of their “softpaq” packages only work on Vista (while they contain the XP drivers as well); at the end of the day, the BIOS is enforcing some stupid policy on the HDA-based soundcard… I was able to get it running by using the devcon.exe command from Microsoft and making it reset the PCI ID of the soundcard at each Windows startup, which makes it work nicely.

Or take Gigabyte, which usually has a decent support for Linux: yesterday I built a computer for a friend of mine, with a Gigabyte GA-970A-UD3 motherboard; he’s running Windows 7 there, but as usual I wanted to write down the list of components and settings with lshw, so I plugged in my usual SysRescueCD thumbdrive and … it didn’t boot. The same goes for the CD-Rom version; FreeDOS and Windows 7 boot cleanly, so my first guess is that there is something wrong, or at least different, in the way Syslinux boots. Contrarily to the kind of replies I received on twitter, I don’t think that Gigabyte is “not supporting Linux” given that they do list Linux support on their website for this board, more likely there is something funky with SysLinux.

But today’s hall of shame entry is quite enraging: Packard Bell (which has been bought by Acer a few years back) has a netbook line that is called “dot”; an acquaintance of mine received a “dot S” device that is actually a DOT_SE3/W-100IT, which comes with 1GB of RAM, and he asked me if I could get more RAM on it. Sure usually I can — in this case the maximum available is 2GB. He brought the device to me and I tried to find how to open it…

There are no instructions, it’s hard to find anything; DuckDuckGo does not find anything useful, while Google’s “did you mean?” feature made it impossible to find something related to SE3, with many more sources for SE2 and simple S instructions. It goes without saying that neither is anywhere near similar to this one. At the end of the day it seems like the only way you have to access the backside panel under which the memory is, is to disassemble almost the whole motherboard. Not going to.

You probably remember my previous notes about Wordpress, FTP and the problem with security. At the end after a (boring) set up session I was able to get vsftpd provide FTPS service, which should be usable both by Wordpress and by Dreamweaver, so that my friend the webmaster can upload through it directly.

This is important because as it happens I have another prospective customer who’s going to run Wordpress, and FTPS now start to look more interesting than SSH, as it doesn’t require me to give shell access to the server either.

Unfortunately I’m a bit worried (maybe more than I should be) for the use of standard passwords rather than certificates or keypairs for authentication. Which meant I went tried to think of other alternatives.. of which there are mostly two: Google Authenticator and YubiKey .

The latter I knew by name already because I proxy-maintain the required software for Brant, and I know it’s outdated already and would require a new maintainer who can deal with those packages – I already posted about hardware-related maintenance for what it’s worth – so it was my first choice: while it meant I had to spend some money, it would have solved my problem and improved Gentoo, even if just for a tiny bit. The price for YubiKey devices is also low enough that, if I felt like providing more FTPS access to customers, I could simply bill it to them without many complaints.

So I went on the manufacturer’s (Yubico’s) website and tried to buy two of them (one for me to test and set up, and one to give my friend to access the server); despite publishing the prices in dollars, they sell through Sweden and UK, which means they are part of EU’s VAT area, and me being a registered business within EU, I should receive a reverse-charge invoice by stating my own VAT ID… never had much of a problem with it, as many of my suppliers are sparse through Europe, I registered for the “foreign-enabled” registry right when I opened business — don’t ask me why Italian (and Spanish as far as I can tell) business owners are not enabled by default to have intra-union suppliers.

Now trouble starts: since, as I just noted, not all VAT IDs are valid to use for intra-union trade, there has to be a way to ensure you’re dealing with an acceptable party. This is implemented through VIES the VAT Information Exchange System which, for what concerns Italian businesses, only tells you a boolean result of valid/invalid (and not the full registration data that most other states seem to provide). I knew VIES from a previous business agreement, but I never cared much. Turns out though that most e-Shops I encountered validate the VAT ID after order completed ­— or in the case of Amazon it seems like they check their internal database as well as VIES.

Yubico instead validates the request through VIES at the time of registration:

VAT Number could not be validated with VIES at this time. This typically happens when the service is under maintenance. Please retry after some time. For urgent orders, please contact order@yubico.com

Considering that the VIES website has a long disclaimer (which I can’t quote here for reasons that will be clear in a moment) stating that they do not guarantee the availability of the service at any time, and only seem to guarantee the validity of the data to the extent that the law ask them to (which probably means “as long as the states’ own databases are correct”), relying on such a service for registration is .. bad.

The VIES website is indeed down since at least 11am today (over four hours ago as I write this); for a moment they also gave me an interesting page (which I forgot to save), telling me that there were too many requests’ failures from “my IP address” … listing an IP address in the 212/8 range — my actual IP address is in the 94/8 range.

What’s the end result here? I’ll probably waste some more time trying to get Google Authenticator; Yubico basically lost a customer and a (possible) contributor by trying and failing to be smarter and won’t have a dedicated maintainer in Gentoo in the near future. It’s sad, because it seems to be easily the most cost- and time-effective solution out there (Google Authenticator is free, but it requires a greater investment of time, and time is money as we all should know).

Okay, I love to rant, so what?

Just the other day I have complained about Rails’s suggestion for world-writable logs and solved it by making it use syslog and now I’m in front of another situation that makes me think that people still don’t know how to stop themselves from creating software that is pretty much insecure by design.

So what’s up? For a customer of mine I ended up having to install a full LAMP stack, rather than my usual LAPR. In particular, this is for a website that will have to run Wordpress. Thankfully, I have ModSecurity to help me out, especially since not even two hours after actually setting up the instance, Spiderlabs announced two more security issues including an extract of their commercial rules.

Anyway, the Wordpress instance will have to be managed/administered by a friend of mine, who has already had some trouble before with a different hoster, where the whole Wordpress instance was injected with tons of malware, so was quite keen on letting me harden the security as much as I could… the problem here is that it seems like there’s not much that I can!

The first problem is that I don’t have a clean way to convert the admin section to forced SSL: not only wp-login.php is outside of the admin subdirectory, but most of Wordpress seem to use fully qualified, absolute URIs rather than relative URLs — such as the ones I’m used with Rails, which in the case both of Typo and Radiant let me restrict the admin/ directory to SSL quite easily. Why is that so important to me? Because I would have used an admin URL outside of the website’s domain for SSL: I don’t own a certificate for the website’s domain, which is not mine, nor I want to add it to the list of aliases of my own box. Oh well for now they’ll live with the “invalid certificate” warning.

Next stop is updating the webapp itself; I was sure at that point that “updating the webapp” meant letting the web server write to the wordpress deployment directory… yes, but that’s just part of it. As it happens, plugins are updated via FTP, like my friend told me.. but not in the sense of “downloaded from an FTP website and written to the filesystem” but the other way around: you have to tell Wordpress how to access its own deployment via FTP. In a clear-text web form. Admittedly, it supports FTPS, but it’s still not very funny.

I’m unsure if it was a good idea on my part to accept hosting Wordpress: we’re talking about installing MySQL, PHP, vsftpd and enabling one more service on the box (vsftp) just to get a blogging platform. Comparatively, Rails look like a lightweight approach.

Well… I finally figured out that the ucb package isn’t installed on Solaris 11 by default (resource). Unfortunately, the Oracle docs are confusing to follow. Here is a cheatsheet for installing the ucb package on your shiny Solaris 11 install.

1. Figure out the IPS installer, read man pages, get frustrated at lack of detail, run to Google.
2. Find the package you want on http://pkg.oracle.com/, in this case compatibility/ucb
3. Add the publisher link to your config, by the way, this link is not documented that I can find so I had to guess and check. A publisher is a package list of sorts, I guess.
   # pkg set-publisher -G '*' -M '*' -g http://pkg.oracle.com/solaris/release solaris
4. Install the package, # pkg install compatibility/ucb

# pkg install compatibility/ucb
Packages to install: 1
Create boot environment: No
Create backup boot environment: No

DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 80/80 0.4/0.4

PHASE ACTIONS
Install Phase 166/166

PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2

1. Behold, that you now have the compatibility libs for software that may need to use them

Whew…now, you might wonder what is so hard about that. Well, traversing Oracle docs is the hard part.

Here are the docs that I had open in my browser, they may or may not help and I fully expect the links to break in the future because Oracle is good at that.

• Copying and Creating Oracle Solaris 11 Package Repositories
• Oracle Solaris 11 Package Repository
• Oracle Solaris 11 Package Management with IPS
• Image Packaging System Man Pages

Seems the patch I committed for the fix was corrupted.  So, I am rebuilding and releasing kernels for 3.2 , 3.1 and 3.0.

Thanks for wired for pointing this out.  I will be removing the ones from yesterday.

The following kernels now contain the fix:

gentoo-sources-3.2.1-r2

gentoo-sources-3.1.10-r1

gentoo-sources-3.0.17-r2

In my previous installment I ranted about. among other things, the way Rails suggests you to keep a world-writeable log file for the production environment. As I said at the end, I planned on looking at the syslogger gem and that was actually quite helpful.

The idea goes like this: by using syslogger you can tell Rails that the logs have to go through the syslog; in my case that means it goes to metalog, which then filters on the webapp names and pushes it to /var/log/rails, taking care of rotating the log as needed (either due to size or time — the former is quite useful to avoid that rogue bots cause a DoS, which happened to me when I was inexperienced with these technologies!). Of course, this only works on Unix, but that’s what I care about anyway.

Beside the placement of the logs, using metalog for me also means I can filter important messages and show them in the important messages’ log rather than being just limited to a hidden log file within the app’s own tree, and also means that I can mix in the messages of all the running applications, rather than having each report to a different file. If I were to use syslog-ng instead, I could easily make it send the logs via network to another box and aggregate all of them there… but I really don’t see the point (yet) for that, and the features that metalog comes with tramp easily the network support.

So how do you achieve this? It’s actually pretty easy. Obviously it starts with installing dev-ruby/syslogger (in Gentoo, through Portage, everywhere else, via gem); then you can configure this very easily on both Rails 2.3 and 3.x series (I have one server running Rails 2.3, the other 3.1… I have yet to set up Typo 6.x, but I’ll probably do that at some point in the near future, although unlikely before FOSDEM).

The trick is all in config/environments/production.rb, where you have to tell Rails to use a custom Logger; there is already an example, commented-out like that refers to the other gem, SyslogLogger, but you should change it to something like this

  config.logger = Syslogger.new("yourappname")

This way you can distinguish each application’s messages in the log. Then in the metalog.conf file you can have:

Rails apps : 
  program_regex = "^(typo|radiant|yourappname)"
  logdir = "/var/log/rails"
  maxfiles = 5
  break = 1

so that everything is then readable as /var/log/rails/current.

I’m not sure how much it impacts performance; I’d be surprised if it decreased them, as metalog also buffers the disk writes, but you never know until you check for sure; in general I still prefer if the (multiple) Rails processes send everything to metalog for my own convenience.

Interestingly, if you have a webapp that does not deal with on-disk files directly, but just with a database, by using syslogger you’re basically limiting the writing to the cache directories only, which is probably a positive note.

Okay, so I decided to start yet. another. new. blog. It’s called “working with teenagers”. I’m reproducing … at least, in some fashion. I wonder if my parents are proud of me. Late at night, they can stay up and say, “this is about as close to grandkids as we’ll get! Pass me some Wheat Thins.” Seems reasonable.

Really, though, since I’m going to school to, you know, do this full-time, I thought it’d be cool to archive my old posts about working with them, and just post stuff to it whenever I feel like it. Like tonight, I just added another one, and I figured, “I should probably go to bed. And also write a blog post and my other blog!” And then my mind went blank after that.

In addition to the archives of stuff on here that you’ve already taken the time to memorize, I’ve added two new posts over there since then. You’ll notice that I’ve refrained from shamelessly using my blog to do some cross-posting mojo to do some self-promotion … at least until tonight. To make it seem like this blog post has actual content, I’ll throw in something slightly more interesting.

I found out recently that I really enjoy bowling. Me and my cousin have been going for a few weekends in a row. We’ve mastered the art of playing 4 games in a row for $10. That’s not bad, considering it’s late Saturday nights. Good times. I’m actually getting better at (since it’s impossible to be worse). The hardest part is getting people to ignore that I’m using an 8 pound ball because I’ll throw out my wrists if I use anything heavier.

Where was I going with all of this. I remember I was playing Skyrim tonight (level 60, yo!), and I was fighting a dragon and trying to eat cheesy nacho goodness at the same time. I kept having to pause my game so I could eat, and I thought to myself, “I can’t pause a nacho.” Words to live by.

In other entertainment, I present to you, the best picture on the internet:

It’s totally legit. They have their own domain and everything: http://thebestpictureontheinternet.com/

I think it’s time to go to bed.

I just released gentoo-sources-3.2.1-r1 for Linux Local Privilege Escalation via SUID /proc/pid/mem .

I plan on creating releases for additional kernels with this patch through the day.

See the link for more info on the privilege escalation.

The following kernel versions contain the patch:

gentoo-sources-3.2.1-r1

gentoo-sources-3.1.10

gentoo-sources-3.0.17-r1

At Sportradar, we have several products where everything is hosted on our servers, but our customers embed the them into their websites. The result is that we concurrently handle the accumulated traffic of all our customers. On a typical Saturday this is a five-digit number of requests per second. In order to handle all this traffic, and more importantly, making it easy to scale up to meet future traffic demands as we sell more products, we have spent quite a bit of time on researching what kind of service infrastructure works best with as little hardware as possible.

The stack I will describe here is not the same as we are using. It is a simplification. Linux provides a lot of buttons to push and knobs to turn that affect performance. But these settings are typically very tied to the workload and very difficult to generalise. We have achieved an understanding, mostly by trial and error, about what works for us, but the same settings will probably not be useful to anyone else.

This article is only concerned about how requests move from your users to the web servers serving content. It does not deal with how to scale the web application itself. I will also not go into much detail about how to configure each of the services mentioned.

Principles

I have to say I am a big fan of the Unix philosophy of using small, specialised services. It is the primary reason I like to use web servers like nginx, which only handles one single task, and why I think using PHP FPM instead of Apache/Mod_PHP is a good idea. Just like with programming, keeping stuff compartmentalised makes debugging easier, it leads to single failing nodes affecting only single services, and it is a whole lot easier to scale where necessary.

All of the machine in this setup are virtualised using Kernel Virtual Machine (KVM), and managed by Ganeti. The cool thing about using Ganeti is that it supports syncing disks to a secondary hypervisor using Distributed Replicated Block Device (DRBD). If any of these nodes fail, they can just be booted on the secondary hypervisor and pick up where the failing node left off. Note that if your application is very CPU bound, I would not use virtualisation. You lose quite a bit of CPU and I/O performance when virtualising.

The stack

Let me start of by presenting the stack. Then I’ll go through each level and give some more thorough explanations later.

1. Gateway
2. SSL termination/proxy
3. HTTP Accelerator
4. Web server/FCGI

Granted, using this many systems require its cost of system administration. But since the nodes individually are so simple, running software upgrades is rather trivial as there are no conflicting dependencies. Using virtual machines also make dist-upgrades trivial. We simply do not ever do it. Instead we fire up a new virtual machine with the newest OS version, configure it, deploy software and do some simple testing, and then just let it be a drop-in replacement of the old node.

The gateway

The purpose of the gateway is to handle routing between Internet and the application-specific subnets. I like using a load balancer like Linux Virtual Server (LVS) for this, because it allows me to scale the layer following horizontally. LVS can basically handle any amount of traffic you throw at it on a single node so there is no need to think about how to add more nodes into this layer. If it really became necessary to do so, and adding more hardware to the existing two nodes would not be possible, DNS round-robin could be a way to achieve a form of load balancing.

Even though I do not find load balancing necessary in this layer, I would still remand redundancy. Not only can nodes fail, but every now and then, I would like to be able to take the load gateway out of production to perform maintenance on it. Redundancy on this level is achieved by using Linux-HA. The simple explanation of what this software suite does is this: If the active node dies, the stand-by node takes over its IP, sends an ARP announcement, and, if configured correctly, resumes the work of the failed node.

SSL termination/proxy

So you may ask “Why do we need dedicated nodes to terminate SSL?”. Firstly it is because both web applications and SSL terminations are typically CPU-bound so you do not want these two parts fighting over resources. Secondly, Varnish, the next service in the stack, does not speak SSL.

This layer need to be scalable horizontally due to the CPU cycles required to terminate SSL. Especially if you allow ciphers using one-time Diffie-Hellman. I always make sure that I have enough nodes on this layer to handle at least a single node failure.

These days I use nginx for this layer, but any kind of light-weight, high-performance web server will do the job. The one thing worth mentioning about using nginx is that it does not (yet) support HTTP 1.1. So no keep-alive connections and no chunked response towards the backend. But since the backend is Varnish, this is not that big of an issue.

HTTP Accellerator

And now for the stack’s super hero: Varnish. It is an HTTP cache server that can handle pretty much any amount of traffic. During my stress testing I have seen Varnish handle thousands of connections on a single CPU core. Therefor I would not worry about scaling this bit horizontally unless you have to cache a huge amount of data.

Another reason for only having a single active node in this layer is that there is a chance for the same page being cached at different times with different contents. If the user continuously hit ‘refresh’ they would end flipping between the two different cached versions making your site look silly.

The redundancy setup is identical to that of the gateway layer.

Web server

In my sketch above, I just added a bunch of Nginx/PHP FPM servers behind the Varnish. This is how the setup would look like in its simplest form, assuming that you do not require cookies, user logins or anything else that require this layer to simulate some form of state.

The important bit is that this layer is easy to scale horizontally. All you need to do is add another server to the director configuration of Varnish. Varnish support several different form of directors, even directors that will help you maintain state. Going into details about this, however, is in itself worthy of an article.

Some final remarks

This setup is a bit simpler than I would put into production, but it contains the essential details. All of the services mentioned are quite trivial to configure and there should be lots of resources online about each of them.

You might or might not remember my fighting with mod_perl and my finding a bug in the handling of logs if Apache’s error log is set to use the syslog interface (which in my case would be metalog). For those wondering the upstream bug is still untouched goes without saying. This should have told me that there aren’t many people using Apache’s syslog support, but sometimes I’m stubborn.

Anyway, yesterday I finally put into so-called “production” the webapp I described last week for handling customers’ computers. I got it working in no time after mongoid started to behave (tests are still restricted, because a couple fail and I’m not sure why — I’ll have to work on that with the next release that require quite fewer hacks to test cleanly). I did encounter a nasty bug in "best_in_place"http://rubygems.org/gems/best_in_place which I ended up fixing in Gentoo even though upstream hasn’t merged my branch yet.

To get it in “production” I simply mean configuring it to run on the twin server of this blog’s, which I’ve been using for another customer as well — and got ready for a third. Since Rails 3.1 was already installed on that box, it was quite easy to move my new app there. All it took was installing the few new gems I needed and…

Well here’s the interesting thing: I didn’t want for my application to run as my user, while obviously I wanted to check out the sources with my user so that I could get it to update with git … how do you do that? Well, Passenger is able to run the application under whatever user owns the config/environment.rb file, so you’d expect it to be able to run under an arbitrary user as well — which is the case, but only if you’re using version 3 (which is not stable in Gentoo as of yet).

So anyway I set up the new passenger to change the user, make public/assets/ and another directory I write to group-writable (the app user and my user are in the same group), and then I’m basically done, I think. I start up and I’m done with it, I think… but the hostnames tell me that “something went wrong”, without any clue as to what.

Okay so the default for Passenger is to not have any log at all, not a problem, I’ll just increase the level to 1 and see the error… or not? I still get no output in Apache’s error log .. which is still set to syslog… don’t tell me… I set Passenger to log to file, and lo and behold it works fine. I wonder if it’s time for me to learn Apache’s API and get to fix both, since it looks like I’m one of the very few people who would like to use syslog as Apache’s error log.

After getting Passenger to finally tell me what’s wrong, I find out both the reason why Rails wasn’t starting (I forgot to enable two USE flags in dev-ruby/barby which I use for generating the QR code on the label), but I also see this:

Rails Error: Unable to access log file. Please ensure that /var/www/${vhost}/log/production.log exists and is chmod 0666. The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
Please note that logging negatively impacts client-side performance. You should set your logging level no lower than :info in production.

What? Rails is really telling its users to create a world writeable log file, when it fails to write to it? Are they freaking kidding me? Is this really a suggestion coming from the developers of a framework for Web Applications which should be security-sensitive? … Okay so one can be smarter than them and do the right thing (in my case make sure that the log file is actually group-writeable) but if this is the kind of suggestions they find proper to tell you, it’s no wonder what happened with Diaspora. So it’s one more reason why Rails shouldn’t be for the faint hearted and that you should pay a very good sysadmin if you want to run a Rails application.

Oh and by the way the cherry on top of this is that instead of just sending the log to stderr, leaving it to Passenger to wrangle – which would have worked out nicely if Passenger had a way to distinguish which app the errors are coming from – Rails also moves the log level to warning, just to spite you. And then tells you that it impacts performances! Ain’t that lovely?

Plan for the day? If I find some extra free time I’d like to give a try and package (not necessarily in this order) syslogger so that the whole production.log thing can go away fast.

Until recently, page margins in LaTeX had more control over me than I had over them. I already heard that package geometry could be of use here, but quick hacks seemed more fun than going through the docs of that package.

I had a closer look now and geometry turned out to be much more convenient than I expected in the end.

The code that I experimented with can be reduced to this snippet:

%% Demo by Sebastian Pipping <sebastian@pipping.org>
%% Released to the public domain
\documentclass[a4paper]{article}
\usepackage[hmargin=2cm,vmargin=1cm]{geometry}
\begin{document}
\rule{\textwidth}{\textheight}
\end{document}

So I am abusing \rule here to draw a filled rectangle that spans the whole content area. I am asking for horizontal margins of 2cm width and vertical ones of 1cm height.

Strangely, the output I received did not match my expectations. Look how much bigger the left margin is than the right one.

It turns out that indentation of the first line of a paragraph is at work here. The insertion of \noindent solved that problem.

Since the general trend on many linux distros is towards requiring /usr to be mounted at boot time, I figured I’d see what it would take to get it working using dracut.

I’ve been messing with dracut for a while, and for some reason it stubbornly refuses to detect my raid devices. The kernel autodetection works fine, but this is disabled when booting from an initramfs. Dracut would timeout and drop me to a dash shell, and if I just typed mdadm -As followed by exit it would boot just fine.

Dracut is using udev to set up raid devices, and obviously that is not working.

Beyond this, I’d like to get my /usr mounted pre-boot, and there is a module called usrmount that purports to do just this. However, it isn’t working in my case because /usr is a bind mount to a subdir on an lvm volume, and it just isn’t figuring that out (it doesn’t even run lvm in the first place despite having the module installed, let alone figuring out what to mount in what order – I suspect the lvm module only works if root is on lvm).

My solution to both problems is to build my own simple dracut module. If you want to try it out:

1. cd /usr/lib/dracut/modules.d/
2. mkdir 91local
3. cat > 91local/module-setup.sh
   #!/bin/bash
   # -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
   # ex: ts=8 sw=4 sts=4 et filetype=sh

   check() {
   return 0
   }

   depends() {
   return 0
   }

   install() {
inst_hook pre-trigger 91 "$moddir/mount-local.sh"
}


4. cat > 91local/mount-local.sh
   #!/bin/sh
   # -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
   # ex: ts=8 sw=4 sts=4 et filetype=sh

   mount_local()
   {
   mdadm -As
   lvm pvscan
   lvm vgscan
   lvm lvscan
   lvm vgchange -ay
   }

   mount_local

Then run dracut to build your initramfs, and it should let mdadm and lvm auto-detect everything before it gets to mounting stuff. You can then use the fstab-sys to mount whatever you need to mount user. However, in your fstab.sys if you’re configuring a bindmount be sure to prepend /sysroot/ before the source directory.
Example fstab.sys:
/dev/vg1/data /data ext4 noatime,user_xattr,barrier=1 0 0
/sysroot/data/usr /usr none bind 0 0
/sysroot/data/var /var none bind 0 0

Hopefully this helps somebody out – the dracut documentation is pretty sparse. In fact, if somebody connected to dracut stumbles upon this I’d be open to a better way of hooking my script – pre-trigger just doesn’t seem right – I’d rather let udev try to do everything first. However, I couldn’t find any way to hook after udev runs but before it bombs out not finding my root device. Suggestions welcome.

1) Roll call

alexxy, jmbsvicetto, dilfridge, johu, mschiff, tampakrap, Thev00d00

2) Electing a new team leader

Since one year is not over yet, it will be skipped for the next meeting.

3) What shall we do with kdepim-4.4

KDEPIM 4.4 is not supported any more by upstream, but on the other hand KDEPIM2 is still too buggy. We had a discussion if we should remove it completely or if we should continue maintain it, despite the compatibility bugs that started to emerge with newer KDE versions. Final decision is that we will continue support it as long it works with newer KDE SC releases. We’ll keep the kdepim-l10n split package to provide the translations for it.

4) kdeenablefinal revisited

Since upstream doesn’t seem to care about it much, plus it doesn’t make much sense now that there are many split tarballs, we decided to remove it the next day after the meeting.

5) phonon-xine removal

KDE upstream acknowledged that this is not maintained anymore. It’s already masked since 2011/12/01. Will be last rited and removed 15 days afterwards.

6) Qt 4.8

We expect no big issues with it. Kdenlive is the only known application that does not build at the moment and will be patched. kde-base/kstyles-4.7.* needs to be rebuilt after the upgrade, which we’ll solve with a combination of revbump/dependencies (otherwise KDE apps using oxygen style crash).

7) Dropping RPATH from installed binaries

Postponed for next meeting, need more info from reavertm and/or hardened herd.

To eselect Boost or not to eselect boost

No final decision was taken, discussion will be moved to -dev mailing list.

9) Bugs

* dev-util/cmake picks always the latest boost. Fix in overlay since 13. Dec. Move to tree? https://bugs.gentoo.org/show_bug.cgi?id=335108

see 8.

* cmake-utils.eclass PREFIX is not defined, any progress? https://bugs.gentoo.org/show_bug.cgi?id=358059

Postponed for next meeting

* Remove hard dep on media-libs/phonon from kde-base/kdelibs https://bugs.gentoo.org/show_bug.cgi?id=356681 https://bugs.gentoo.org/show_bug.cgi?id=388041

Although it is possible to build kdelibs against qt-phonon, it is not recommended by upstream. Decision postponed for next meeting.

* Eclass problem with handbook without LINGUAS. https://bugs.gentoo.org/show_bug.cgi?id=372457

Needs more analysis. Postponed.

* MacOSX request for cmake-utils.eclass: Remove force of  CMAKE_BUILD_WITH_INSTALL_RPATH=TRUE https://bugs.gentoo.org/show_bug.cgi?id=398437

That was a request by the Gentoo Prefix team, and got accepted

* Revise the change “semantic-desktop? -> semantic-desktop=”. Why was the change needed. https://bugs.gentoo.org/show_bug.cgi?id=396491

We had split opinions on this. Skipped for next meeting, as we need reavertm’s input on this.

10) Open floor

• Tampakrap will make a KDE SC 4.8 release party in Prague, more info coming soon.
• Qt meeting on Thursday 26th Jan.
• See you at fosdem

(I realize that I go on 7-8 skydiving “trips” per year and I often don’t write about them. It may be fun to read about the special events later, so I’m going to try my best to publicly write about my adventures. For an aggregation of random skydiving blogs, check out planetskydive.net)

A few weeks ago, I returned from a 12 day skydiving trip to Skydive Arizona. The highlights of this trip were 2011 Holiday Boogie, freefly load organizers, an hour of tunnel time, getting food poisoning, and a CRW Bigway camp. All while meeting new and old friends. This was my first far-away skydiving trip that I went on by myself, it takes abit to figure out who to jump with but just introducing yourself to the load organizers helps with that. Soon, they are seeking you out to jump because it is their job to encourage you to spend money

So, a great success overall. I made 53 skydives in 10 days (minus 2 days for the food poisoning incident) and had a great time. There are pictures of the CRW camp, here. A couple of my favorites are these…

And of course, my cutaway…

This release features build system changes and fixes. Please check the change log for details. This release is both source- and binary compatible.

Download:
https://sourceforge.net/projects/uriparser/files/Sources/0.7.6/

Change log:
http://uriparser.git.sourceforge.net/git/gitweb.cgi?p=uriparser/uriparser;a=blob;f=ChangeLog

To my European readers: if you care about the impact of social technologies like Git (and GitHub) & how they’re transforming software development, or the impact of social technology on communities, and you enjoy good beer, you need to be at Monki Gras. I just posted over at my RedMonk blog about how the previous conference in the series, Monktoberfest, was the best conference of my life. And I’ve been to many.

Monki Gras is Feb. 1–2 in London. The timing’s perfect to stop by just before FOSDEM (and that’s exactly what I’m doing). Registration is dirt-cheap, speakers are universally top-notch, and you’ll also get some world-class beers in the package.

Yesterday (2012/01/16 20:00 UTC) we had the first Gentoo KDE team meeting this year. The meeting happened in #gentoo-meetings on freenode.

• Participants: alexxy, dilfridge, jmbsvicetto, johu, mschiff, tampakrap, Thev00d00
• Agenda
• Log
• Lead election is delayed, because 12 months not over
• We keep kdepim-4.4 in tree as long as it works and provide kdepim-l10n package
• Kdeenablefinal build feature will be removed today
• Phonon xine backend will be removed in 15 days
• We expect no big issues with Qt 4.8,  only kdenlive is not building at the moment
• eselect boost vs. latest boost is not a Gentoo KDE scope only issue, we will move the discussion to the gentoo-dev mailing list
• … read log or wait for the full summary by tampakrap
• My netbook had a kernel panic while the meeting  :-/

I’ve been meaning to try this for a while, and we’ve heard a number of requests from the community as well. Recently, I got some time here at Collabora to give it a go — that is, to get PulseAudio running on an Android device and see how it compares with Android’s AudioFlinger.

The Contenders

Let’s introduce our contenders first. For those who don’t know, PulseAudio is pretty much a de-facto standard part of the Linux audio stack. It sits on top of ALSA which provides a unified way to talk to the audio hardware and provides a number of handy features that are useful on desktops and embedded devices. I won’t rehash all of these, but this includes a nice modular framework, a bunch of power saving features, flexible routing, and lots more. PulseAudio runs as a daemon, and clients usually use the libpulse library to communicate with it.

In the other corner, we have Android’s native audio system — AudioFlinger. AudioFlinger was written from scratch for Android. It provides an API for playback/recording as well as a control mechanism for implementing policy. It does not depend on ALSA, but instead allows for a sort of HAL that vendors can implement any way they choose. Applications generally play audio via layers built on top of AudioFlinger. Even if you write a native application, it would use OpenSL ES implementation which goes through AudioFlinger. The actual service runs as a thread of the mediaserver daemon, but this is merely an implementation detail.

Note: all my comments about AudioFlinger and Android in general are based on documentation and code for Android 4.0 (Ice Cream Sandwich).

The Arena

My test-bed for the tests was the Galaxy Nexus running Android 4.0 which we shall just abbreviate to ICS. I picked ICS since it is the current platform on which Google is building, and hopefully represents the latest and greatest in AudioFlinger development. The Galaxy Nexus runs a Texas Instruments OMAP4 processor, which is also really convenient since this chip has pretty good support for running stock Linux (read on to see how useful this was).

Preparations

The first step in getting PulseAudio on Android was deciding between using the Android NDK like a regular application or integrate into the base Android system. I chose the latter — even though this was a little more work initially, it made more sense in the long run since PulseAudio really belongs to the base-system.

The next task was to get the required dependencies ported to Android. Fortunately, a lot of the ground work for this was already done by some of the awesome folks at Collabora. Derek Foreman’s androgenizer tool is incredibly handy for converting an autotools-based build to Android–friendly makefiles. With Reynaldo Verdejo and Alessandro Decina’s prior work on GStreamer for Android as a reference, things got even easier.

The most painful bit was libltdl, which we use for dynamically loading modules. Once this was done, the other dependencies were quite straightforward to port over. As a bonus, the Android source already ships an optimised version of Speex which we use for resampling, and it was easy to reuse this as well.

As I mentioned earlier, vendors can choose how they implement their audio abstraction layer. On the Galaxy Nexus, this is built on top of standard ALSA drivers, and the HAL talks to the drivers via a minimalist tinyalsa library. My first hope was to use this, but there was a whole bunch of functions missing that PulseAudio needed. The next approach was to use salsa-lib, which is a stripped down version of the ALSA library written for embedded devices. This too had some missing functions, but these were fewer and easy to implement (and are now upstream).

Now if only life were that simple. :) I got PulseAudio running on the Galaxy Nexus with salsa-lib, and even got sound out of the HDMI port. Nothing from the speakers though (they’re driven by a TI twl6040 codec). Just to verify, I decided to port the full alsa-lib and alsa-utils packages to debug what’s happening (by this time, I’m familiar enough with androgenizer for all this to be a breeze). Still no luck. Finally, with some pointers from the kind folks at TI (thanks Liam!), I got current UCM configuration files for OMAP4 boards, and some work-in-progress patches to add UCM support to PulseAudio, and after a couple of minor fixes, wham! We have output. :)

(For those who don’t know about UCM — embedded chips are quite different from desktops and expose a huge amount of functionality via ALSA mixer controls. UCM is an effort to have a standard, meaningful way for applications and users to use these.)

In production, it might be handy to write light-weight UCM support for salsa-lib or just convert the UCM configuration into PulseAudio path/profile configuration (bonus points if it’s an automated tool). For our purposes, though, just using alsa-lib is good enough.

To make the comparison fair, I wrote a simple test program that reads raw PCM S16LE data from a file and plays it via the AudioTrack interface provided by AudioFlinger or the PulseAudio Asynchronous API. Tests were run with the brightness fixed, wifi off, and USB port connected to my laptop (for adb shell access).

All tests were run with the CPU frequency pegged at 350 MHz and with 44.1 and 48 kHz samples. Five readings were recorded, and the median value was finally taken.

Round 1: CPU

First, let’s take a look at how the two compare in terms of CPU usage. The numbers below are the percentage CPU usage taken as the sum of all threads of the audio server process and the audio thread in the client application using top (which is why the granularity is limited to an integer percentage).

At 44.1 kHz, the two are essentially the same. Both cases are causing resampling to occur (the native sample rate for the device is 48 kHz). Resampling is done using the Speex library, and we’re seeing minuscule amounts of CPU usage even at 350 MHz, so it’s clear that the NEON optimisations are really paying off here.

The astute reader would have noticed that since the device’ native sample rate is 48 kHz, the CPU usage for 48 kHz playback should be less than for 44.1 kHz. This is true with PulseAudio, but not with AudioFlinger! The reason for this little quirk is that AudioFlinger provides 44.1 kHz samples to the HAL (which means the stream is resampled there), and then the HAL needs to resample it again to 48 kHz to bring it to the device’ native rate. From what I can tell, this is a matter of convention with regards to what audio HALs should expect from AudioFlinger (do correct me if I’m mistaken about the rationale).

So round 1 leans slightly in favour of PulseAudio.

Round 2: Memory

Comparing the memory consumption of the server process is a bit meaningless, because the AudioFlinger daemon thread shares an address space with the rest of the mediaserver process. For the curious, the resident set size was: AudioFlinger — 6,796 KB, PulseAudio — 3,024 KB. Again, this doesn’t really mean much.

We can, however, compare the client process’ memory consumption. This is RSS in kilobytes, measured using top.

The memory consumption is comparable between the two, but leans in favour of AudioFlinger.

Round 3: Power

I didn’t have access to a power monitor, so I decided to use a couple of indirect metrics to compare power utilisation. The first of these is PowerTOP, which is actually a Linux desktop tool for monitoring various power metrics. Happily, someone had already ported PowerTOP to Android. The tool reports, among other things, the number of wakeups-from-idle per second for the processor as a whole, and on a per-process basis. Since there are multiple threads involved, and PowerTOP’s per-process measurements are somewhat cryptic to add up, I used the global wakeups-from-idle per second. The “Idle” value counts the number of wakeups when nothing is happening. The actual value is very likely so high because the device is connected to my laptop in USB debugging mode (lots of wakeups from USB, and the device is prevented from going into a full sleep).

The second, similar, data point is the number of interrupts per second reported by vmstat. These corroborate the numbers above:

PulseAudio’s power-saving features are clearly highlighted in this comparison. AudioFlinger causes about three times the number of wakeups per second that PulseAudio does. Things might actually be worse on older hardware with less optimised drivers than the Galaxy Nexus (I’d appreciate reports from running similar tests on a Nexus S or any other device with ALSA support to confirm this).

For those of you who aren’t familiar with PulseAudio, the reason we manage to get these savings is our timer-based scheduling mode. In this mode, we fill up the hardware buffer as much as possible and go to sleep (disabling ALSA interrupts while we’re at it, if possibe). We only wake up when the buffer is nearing empty, and fill it up again. More details can be found in this old blog post by Lennart.

Round 4: Latency

I’ve only had the Galaxy Nexus to actually try this out with, but I’m pretty certain I’m not the only person seeing latency issues on Android. On the Galaxy Nexus, for example, the best latency I can get appears to be 176 ms. This is pretty high for certain types of applications, particularly ones that generate tones based on user input.

With PulseAudio, where we dynamically adjust buffering based on what clients request, I was able to drive down the total buffering to approximately 20 ms (too much lower, and we started getting dropouts). There is likely room for improvement here, and it is something on my todo list, but even out-of-the-box, we’re doing quite well.

Round 5: Features

With the hard numbers out of the way, I’d like to talk a little bit about what else PulseAudio brings to the table. In addition to a playback/record API, AudioFlinger provides mechanism for enforcing various bits of policy such as volumes and setting the “active” device amongst others. PulseAudio exposes similar functionality, some as part of the client API and the rest via the core API exposed to modules.

From SoC vendors’ perspective, it is often necessary to support both Android and standard Linux on the same chip. Being able to focus only on good quality ALSA drivers and knowing that this will ensure quality on both these systems would be a definite advantage in this case.

The current Android system leaves power management to the audio HAL. This means that each vendor needs to implement this themselves. Letting PulseAudio manage the hardware based on requested latencies and policy gives us a single point of control, greatly simplifying the task of power-management and avoiding code duplication.

There are a number of features that PulseAudio provides that can be useful in the various scenarios where Android is used. For example, we support transparently streaming audio over the network, which could be a handy way of supporting playing audio from your phone on your TV completely transparently and out-of-the-box. We also support compressed formats (AC3, DTS, etc.) which the ongoing Android-on-your-TV efforts could likely take advantage of.

Edit: As someone pointed out on LWN, I missed one thing — AudioFlinger has an effect API that we do not yet have in PulseAudio. It’s something I’d definitely like to see added to PulseAudio in the future.

Ding! Ding! Ding!

That pretty much concludes the comparison of these two audio daemons. Since the Android-side code is somewhat under-documented, I’d welcome comments from readers who are familiar with the code and history of AudioFlinger.

I’m in the process of pushing all the patches I’ve had to write to the various upstream projects. A number of these are merely build system patches to integrate with the Android build system, and I’m hoping projects are open to these. Instructions on building this code will be available on the PulseAudio Android wiki page.

For future work, it would be interesting to write a wrapper on top of PulseAudio that exposes the AudioFlinger audio and policy APIs — this would basically let us run PulseAudio as a drop-in AudioFlinger replacement. In addition, there are potential performance benefits that can be derived from using Android-specific infrastructure such as Binder (for IPC) and ashmem (for transferring audio blocks as shared memory segments, something we support on desktops using the standard Linux SHM mechanism which is not available on Android).

If you’re an OEM who is interested in this work, you can get in touch with us — details are on the Collabora website.

I hope this is useful to some of you out there!

I’ve spent the first week of the year on vacation with some friends. The second week of the year has been mixed between going on with the jobs I should have gotten working already, fighting a bad case of cold, and getting insulted by a customer of mine for actually having gotten real vacation time for once in two years. More to the point: said customer doesn’t actually pay me overtime, or actually at all for the support.

Tonight I wanted to relax and think about my own needs. Not personal needs, alas, but at least needs for my work to become easier. Since I haven’t made any progress at all regarding RT I decided to look into a different need of mine: cataloguing customers’ computers.

I originally simply kept a file listing the computers I set up for customers — then I started getting more customers, and sometimes getting a computer back after many months since last time. And I started forgetting which computer was which. Nowadays I have 79 computers on my “database” (which is just a git repository with a bunch of HTML files as well as lshw dumps), without counting those that have been dismissed.

To recognise the computers, I started printing labels with a QR Code on them, which contains the URL of the computer’s HTML file on my website (password-protected). My original method required me to feed a multi-label A4 sheet into my laser printer and print one, two or three labels out on that… but it turned out to be a waste of time and of money in sheets, given that most of the time I ended up wasting half of it, as the printer refused to print aligned more than half the time. I’ve since bought a Dymo label printer, which is why you’ll find their drivers in Portage maintained by yours truly — the nice thing about Dymo’s label printers is that their drivers are fully GPL-2, while as far as I can tell both Zebra and Brother have binary blobs, that make them unsuitable for use on amd64-based systems.

As you can tell, there are a few things that I did in Gentoo that relate to this little “database” of mine: the lshw fixes to try getting it back into SysRescueCD (it’s still not there — and I lost the password for my account on their forums), the Dymo drivers noted above, and dev-ruby/barby which is a quite interesting library that allows you to generate almost any kind of barcode. And now it’s time of MongoDB Ruby libraries as I’m trying to write an actual web application to manage the “database” and make it a real database.

Today’s achievement is big: I finally got Rails (3.1) to play nice with MongoDB. Not using MongoMapper, the author of which, as I already talked about I would prefer not having much to discuss with. But thanks to Mauro I got pointed at Mongoid which is a much more well developed alternative.

Okay sure there are quite a few things to kink out in the packaging of Mongoid – for instance the fact that the gem packages a Rakefile that relies on a (missing) Gemfile, or the fact that two out of three rspec targets in said Rakefile fail, one of which by crashing the interpeter – but at least their unit-tests work, and the code works as intended when loaded it up. Which is more I can say about MongoMapper.

Oh and it doesn’t seem to require extra code to be added just to work correctly with Passenger.

The only problem I have now is fixing up one side issue: how do I print the labels once I load this into my webserver? I could download the PDF I use to print the label and then print that.. but it’s a bit of a time-waster. Of course both the server and Yamato (where the label printer is connected) are IPv6-enabled and .. well, the IPP protocol used by CUPS is fine to be used over the internet, as it can use SSL encryption. Which yes, means that I’ll be setting up a web application … that calls home to print a label, how crazy is that?

My only issue with this is that I’d rather not install cups on the webserver (especially since there is currently no way to just build the client side of it, which would be the only part of it I would need on the server — yeah I know, it’s funky), so I can’t just call lpr mylabel.pdf… and as far as I can tell, the only way to access IPP from Ruby is one of the many CUPS library bindings available as gems, which are all 0.0 versions, and do not inspire me the least. Since IPP is based off HTTP, I would have expected more implementations of it, to be honest.

Possibly, it should be possible to extend some HTTP Ruby library to send IPP requests as well; for what I’m concerned, I’d just need the “Print-Job” method to be implemented, which would allow me to send the PDF file to be printed with the default options. I guess I’ll resolve that bit once I’m done with the rest of my application, though.

I’m no fan of initramfs. All my systems boot up just fine without it, so I often see it as an additional layer of obfuscation. But there are definitely cases where initramfs is needed, and from the looks of it, we might be needing to push out some documentation and support for initramfs. Since my primary focus is to look at a hardened system, I started playing with initramfs together with Gentoo Hardened, grSecurity and SELinux. And what a challenge it was…

But first, a quick introduction to initramfs. The Linux kernel supports initrd images for quite some time. These images are best seen as loopback-mountable images containing a whole file system that the Linux kernel boots as the root device. On this initrd image, a set of tools and scripts then prepare the system and finally switch towards the real root device. The initrd feature was often used when the root device is a network-mounted location or on a file system that requires additional activities (like an encrypted file system or even on LVM. But it also had some difficulties with it.

Using a loopback-mountable image means that this is seen as a full device (with file system on it), so the Linux kernel also tries caching the files on it, which leads to some unwanted memory consumption. It is a static environment, so it is hard to grow or shrink it. Every time an administrator creates an initrd, he needs to carefully design (capacity-wise) the environment not to request too much or too little memory.

Enter initramfs. The concept is similar: an environment that the Linux kernel boots as a root device which is used to prepare for booting further from the real root file systems. But it uses a different approach. First of all, it is no longer a loopback-mountable image, but a cpio archive that is used on a tmpfs file system. Unlike initrd, tmpfs can grow or shrink as necessary, so the administrator doesn’t need to plan the capacity of the image. And because it is a tmpfs file system, the Linux kernel doesn’t try to cache the files in memory (as it knows they already are in memory).

There are undoubtedly more advantages to initramfs, but let’s stick to the primary objective of this post: talk about its implementation on a hardened system.

I started playing with dracut, a tool to create initramfs archives which is seen as a widely popular implementation (and suggested on the gentoo development mailinglist). It uses a simple, modular approach to building initramfs archives. It has a base, which includes a small init script and some device handling (based on udev), and modules that you can add depending on your situation (such as adding support for RAID devices, LVM, NFS mounted file systems etc.)

On a SELinux system (using a strict policy, enforcing mode) running dracut in the sysadm_t domain doesn’t work, so I had to create a dracut_t domain (which has been pushed to the Portage tree yesterday). But other than that, it is for me sufficient to call dracut to create an initramfs:

# dracut -f "" 3.1.6-hardened

My grub then has an additional set of lines like so:

title Gentoo Linux Hardened (initramfs)
root (hd0,0)
kernel /boot/vmlinuz-3.1.6-hardened root=/dev/vda1 console=ttyS0 console=tty0
initrd /boot/initramfs-3.1.6-hardened.img

Sadly, the bugger didn’t boot. The first problem I hit was that the Linux kernel I boot has chroot restrictions in it (grSecurity). These restrictions further tighten chroot environments so that it is much more difficult to “escape” a chroot. But dracut, and probably all others, use chroot to further prepare the bootup and eventually switch to the chrooted environment to boot up further. Having the chroot restrictions enabled effectively means that I cannot use initramfs environments. To work around, I enabled sysctl support for all the chroot restrictions and made sure that their default behavior is to be disabled. Then, when the system boots up, it enables the restrictions later in the boot process (through the sysctl.conf settings) and then locks these settings (thanks to grSecurity’s grsec_lock feature) so that they cannot be disabled anymore later.

But no, I did get further, up to the point that either the openrc init is called (which tries to load in the SELinux policy and then breaks) or that the initramfs tries to load the SELinux policy – and then breaks. The problem here is that there is too much happening before the SELinux policy is loaded. Files are created (such as device files) or manipulated, chroots are prepared, udev is (temporarily) ran, mounts are created, … all before a SELinux policy is loaded. As a result, the files on the system have incorrect contexts and the moment the SELinux policy is loaded, the processes get denied all access and other privileges they want against these (wrongly) labeled files. And since after loading the SELinux policy, the process runs in kernel_t domain, it doesn’t have the privileges to relabel the entire system, let alone call commands.

This is currently where I’m stuck. I can get the thing boot up, if you temporarily work in permissive mode. When the openrc init is eventually called, things proceed as usual and the moment udev is started (again, now from the openrc init) it is possible to switch to enforcing mode. All processes are running by then in the correct domain and there do not seem to be any files left with wrong contexts (since the initramfs is not reachable anymore and the device files in /dev are now set again by udev which is SELinux aware.

But if you want to boot up in enforcing straight away, there are still things to investigate. I think I’ll need to put the policy in the initramfs as well (which has the huge downside that every update on the policy requires a rebuild of the initramfs as well). In that case I can load the policy early up the chain and have the initramfs work further running in an enforced situation. Or I completely regard the initramfs as an “always trusted” environment and wait for openrc’s init to load the SELinux policy. In that case, I need to find a way to relabel the (temporarily created) /dev entries (like console, kmsg, …) before the policy is loaded.

Definitely to be continued…

As known as #335108. This is (was) a long term bug in Gentoo KDE scope. The problem is that if you have two or more different boost versions installed, the latest version will be used at build time, regardless which version is (e)selected. Real world example we have boost  1.46.1 and 1.47.0 installed selected the 1.46 slot, the 1.47 slot would be used at build:

$ eselect boost list
Available boost versions:
 [1]   boost-1.46/default *
 [2]   boost-1.47/default

Last night i patched dev-util/cmake-2.8.6 successfully and made the revision bump today in the kde-overlay. So please test =dev-util/cmake-2.8.6-r5, in the case your maintained package is cmake based and needs dev-util/boost at build time. You should test at least with two different boost versions and of course switch between those to check that the selected version is used.

I bumped dev-util/cmake-2.8.7 in the overlay too. The patch is also included in this version.

Start your engines…

LWN.net brought my attention to this: Microsoft confirms UEFI fears, locks down ARM devices

Dass man RFID-Chips in Kleidung einbauen kann, war mir klar, aber dass es wirklich gemacht wird, hat mich dann doch ziemlich vom Hocker geworfen:

FoeBuD enttarnt RFID-Chips in Kleidung

OpenStack Logo

Last week I wrote a little bit about OpenStack, what it is good for and that I’ve been working on appliance that you can test. I mentioned appliance to make it easier for people to test OpenStack and play with that. Any feedback is of course appreciated. Since then appliance got updated, because something others have been working on. Although there was Xmass and most of us has other things to worry about enjoy, there has been some progress in other parts of our OpenStack Milestone.
While I was fighting to get OpenStack appliance working, Bernhard Wiedemann was working on other way how to make it more convenient for you to try OpenStack on openSUSE. He wrote a script, that makes the whole OpenStack demo setup much easier. So if you don’t want to use appliance and instead would prefer to just get OpenStack working on your existing setup, you can do it pretty easily. Everything is documented on the wiki. But as you are already reading this, I’ll make it even easier for you and will putt a quick how to here as well.
What do you need? You need some packages, right? First of all you need to add repository with the last stable release. This repository contains fixes, patterns and much more as you’ll see later. You can do it by typing:

zypper ar -f \
 obs://Virtualization:/Cloud:/OpenStack:/Diablo/openSUSE_12.1 \
 OpenStack

Now what about packages? Easy, we’ve got patterns for you. So you can do just

zypper in -t pattern openstack-controller openstack-compute-node

And you are ready to go. Almost. You need to configure it. That takes some time and can be pretty complicated. But hey, I said it is easy now, haven’t I? So it is. You just need to install one more package! So type in

openstack-quickstart

This will install few scripts that will help you create your demo setup. Now all you have to do is run yet another command. I know, it’s getting complicated, we are now at four commands, but all of them are quite easy, right? Bear with me, we are almost there… The last command you need is

openstack-quickstart-demosetup

This will take care of everything you need. Unless you have MySQL root password set, it will go ahead and configure everything and when it ends, you can just simply point your browser to http://127.0.0.1 and play with the dashboard using login admin and password openstack. That was easy, wasn’t it?

Very few of you probably remember that over two years ago, in October 2009, I did some investigative work on Portage Tree’s overhead to show just how much space was going to be wasted with small files on filesystems with too big block sizes.

It wasn’t the only time I noted that while for things like Portage, and likely your operating system’s file, it makes sense to have smaller-than-page-size blocks, it doesn’t seem as smart to do the same for bigger files such as music and video. At the time I noted that HFS+ somehow supported 64KiB blocks with the Linux driver – a driver that is very much unstable and often times unusable – while XFS refuses to play well with similarly-sized blocks, even though it is designed to support them.

I’ve read many people complaining that I didn’t know what I was talking about when I called for bigger block sizes for Linux’s filesystems. Many people insisted that the presence of extents in ext4 made it completely moot to have bigger block sizes. If that’s so, I wonder why ext4 now implements bigalloc which is basically a trick to allow bigger block cluster sizes.

I read about it, with the release announcement of kernel 3.2, while I was on vacation and I just couldn’t wait to try it out with some of my filesystems. Luckily I tried it with the least important one, though, as it’s far from being mature for using.

The current implementation does not support online resizing, so you’re supposed to use resize2fs with the unmounted filesystem … too bad that it fails to run entirely when using the latest version of e2fsprogs. Oh and don’t forget that the switch to turn on bigalloc is not documented anywhere yet.

So it is to be expected given that it’s a very new feature, but I wonder why half the fuss about 3.2 release was about a feature … that definitely is not ready for prime time even in testing ground. I just hope that work toward this kind of features will also mean that XFS will gain support for 64KiB blocks, which I would prefer to ext4’s 1MiB clusters in the first place.

Also I would like to point out one thing for those of you who wish to use this feature on volumes shared with Samba to OS X hosts: you’ll end up with tons of space wasted to .DS_Store files unless the inline data feature is also used, and the inode size is increased. On my filesystems, .DS_Store files weight between 741 bytes to 14KiB… I thought I configured Samba to use extended attributes to store the data instead of using external files, but for what I gathered on the Netatalk mailing list recently, this conflicts with the size limit applied to EAs on ext4… I guess this is another of those things that really need some tweaking to get right.

This is just a short post to let my followers know that I’ll be at FOSDEM next month. I’ve booked the flight back in September and I booked the hotel yesterday, so it’s all set. I just hope not to get lost through Bruxelles.

The only reason why I’m posting this is, actually, that I need some suggestion from somebody who knows Belgium: both my phone operators lack dedicated roaming up there, so I’ll probably end up with an hefty bill waiting for me back home. Given in Italy you really can’t get a local pre-paid SIM to user your phone if you’re a tourist, I’m not sure if the same holds true in Belgium. And most importantly, whether I could re-use such a SIM over the years (as I plan on coming to FOSDEM with regularity, if I survive the trip alone this time).

At any rate, if you want to discuss anything in person, I’ll be the guy with the strange hat and the purse satchel (geek points for getting the reference), hanging around with the Gentoo or libav folks.