EDIT: okay, they just released v3.1.1 so here it goes on portage as well !

highlights

• relax validation of x-match binding to headers exchange for compatibility with brokers < 3.1.0
• fix bug in ack handling for transactional channels that could cause queues to crash
• fix race condition in cluster autoheal that could lead to nodes failing to re-join the cluster

3.1.1 changelog is here.

I’ve bumped the rabbitMQ message queuing server on portage. This new version comes with quite a nice bunch of bugfixes and features.

highlights

• eager synchronisation of slaves by policy (manual & automatic)
• cluster “autoheal” mode to automatically choose nodes to restart when a partition has occurred
• cluster “pause minority” mode to prefer partition tolerance over availability
• improved statistics (including charts) in the management plugin
• quite a bunch of performance improvements
• some nice memory leaks fixes

Read the full changelog.

To work on SELinux policies, I use a couple of functions that I can call on the shell (command line): seshowif, sefindif, seshowdef and sefinddef. The idea behind the methods is that I want to search (find) for an interface (if) or definition (def) that contains a particular method or call. Or, if I know what the interface or definition is, I want to see it (show).

For instance, to find the name of the interface that allows us to define file transitions from the postfix_etc_t label:

$ sefindif filetrans.*postfix_etc
contrib/postfix.if: interface(`postfix_config_filetrans',`
contrib/postfix.if:     filetrans_pattern($1, postfix_etc_t, $2, $3, $4)

Or to show the content of the corenet_tcp_bind_http_port interface:

$ seshowif corenet_tcp_bind_http_port
interface(`corenet_tcp_bind_http_port',`
        gen_require(`
                type http_port_t;
        ')

        allow $1 http_port_t:tcp_socket name_bind;
        allow $1 self:capability net_bind_service;
')

For the definitions, this is quite similar:

$ sefinddef socket.*create
obj_perm_sets.spt:define(`create_socket_perms', `{ create rw_socket_perms }')
obj_perm_sets.spt:define(`create_stream_socket_perms', `{ create_socket_perms listen accept }')
obj_perm_sets.spt:define(`connected_socket_perms', `{ create ioctl read getattr write setattr append bind getopt setopt shutdown }')
obj_perm_sets.spt:define(`create_netlink_socket_perms', `{ create_socket_perms nlmsg_read nlmsg_write }')
obj_perm_sets.spt:define(`rw_netlink_socket_perms', `{ create_socket_perms nlmsg_read nlmsg_write }')
obj_perm_sets.spt:define(`r_netlink_socket_perms', `{ create_socket_perms nlmsg_read }')
obj_perm_sets.spt:define(`client_stream_socket_perms', `{ create ioctl read getattr write setattr append bind getopt setopt shutdown }')

$ seshowdef manage_files_pattern
define(`manage_files_pattern',`
        allow $1 $2:dir rw_dir_perms;
        allow $1 $3:file manage_file_perms;
')

I have these defined in my ~/.bashrc (they are simple functions) and are used on a daily basis here ;-) If you want to learn a bit more on developing SELinux policies for Gentoo, make sure you read the Gentoo Hardened SELinux Development guide.

It is a common request in squid to have it block downloading certain files based on their extension in the url path. A quick look at google’s results on the subject apparently gives us the solution to get this done easily by squid.

The common solution is to create an ACL file listing regular expressions of the extensions you want to block and then apply this to your http_access rules.

blockExtensions.acl

\.exe$

squid.conf

acl blockExtensions urlpath_regex -i "/etc/squid/blockExtensions.acl"

[...]

http_access allow localnet !blockExtensions

Unfortunately this is not enough to prevent users from downloading .exe files. The mistake here is that we assume that the URL will strictly finish by the extension we want to block, consider the two examples below :

http://download.com/badass.exe     // will be DENIED as expected

http://download.com/badass.exe?    // WON'T be denied as it does not match the regex !

Squid uses the extended regex processor which is the same as egrep. So we need to change our blockExtensions.acl file to handle the possible ?whatever string which may be trailing our url_path. Here’s the solution to handle all the cases :

blockExtensions.acl

\.exe(\?.*)?$
\.msi(\?.*)?$
\.msu(\?.*)?$
\.torrent(\?.*)?$

You will still be hated for limiting people’s need to download and install shit on their Windows but you implemented it the right way and no script kiddie can brag about bypassing you

There has been a few posts already on the local Linux kernel privilege escalation, which has received the CVE-2013-2094 ID. arstechnica has a write-up with links to good resources on the Internet, but I definitely want to point readers to the explanation that Brad Spengler made on the vulnerability.

In short, the vulnerability is an out-of-bound access to an array within the Linux perf code (which is a performance measuring subsystem enabled when CONFIG_PERF_EVENTS is enabled). This subsystem is often enabled as it offers a wide range of performance measurement techniques (see its wiki for more information). You can check on your own system through the kernel configuration (zgrep CONFIG_PERF_EVENTS /proc/config.gz if you have the latter pseudo-file available – it is made available through CONFIG_IKCONFIG_PROC).

The public exploit maps memory in userland, fills it with known data, then triggers an out-of-bound decrement that tricks the kernel into decrementing this data (mapped in userland). By looking at where the decrement occurred, the exploit now knows the base address of the array. Next, it targets (through the same vulnerability) the IDT base (Interrupt Descriptor Table) and targets the overflow interrupt vector. It increments the top part of the address that the vector points to (which is 0xffffffff, becoming 0×00000000 thus pointing to the userland), maps this memory region itself with shellcode, and then triggers the overflow. The shell code used in the public exploit modifies the credentials of the current task, sets uid/gid with root and gives full capabilities, and then executes a shell.

As Brad mentions, UDEREF (an option in a grSecurity enabled kernel) should mitigate the attempt to get to the userland. On my system, the exploit fails with the following (start of) oops (without affecting the system further) when it tries to close the file descriptor returned from the syscall that invokes the decrement:

[ 1926.226678] PAX: please report this to pageexec@freemail.hu
[ 1926.227019] BUG: unable to handle kernel paging request at 0000000381f5815c
[ 1926.227019] IP: [] sw_perf_event_destroy+0x1a/0xa0
[ 1926.227019] PGD 58a7c000 
[ 1926.227019] Thread overran stack, or stack corrupted
[ 1926.227019] Oops: 0002 [#4] PREEMPT SMP 
[ 1926.227019] Modules linked in: libcrc32c
[ 1926.227019] CPU 0 
[ 1926.227019] Pid: 4267, comm: test Tainted: G      D      3.8.7-hardened #1 Bochs Bochs
[ 1926.227019] RIP: 0010:[]  [] sw_perf_event_destroy+0x1a/0xa0
[ 1926.227019] RSP: 0018:ffff880058a03e08  EFLAGS: 00010246
...

The exploit also finds that the decrement didn’t succeed:

test: semtex.c:76: main: Assertion 'i<0x0100000000/4' failed.

A second mitigation is that KERNEXEC (also offered through grSecurity) which prevents the kernel from executing data that is writable (including userland data). So modifying the IDT would be mitigated as well.

Another important mitigation is TPE – Trusted Path Execution. This feature prevents the execution of binaries that are not located in a root-owned directory and owned by a trusted group (which on my system is 10 = wheel). So users attempting to execute such code will fail with a Permission denied error, and the following is shown in the logs:

[ 3152.165780] grsec: denied untrusted exec (due to not being in trusted group and file in non-root-owned directory) of /home/user/test by /home/user/test[bash:4382] uid/euid:1000/1000 gid/egid:100/100, parent /bin/bash[bash:4352] uid/euid:1000/1000 gid/egid:100/100

However, even though a nicely hardened system should be fairly immune against the currently circling public exploit, it should be noted that it is not immune against the vulnerability itself. The methods above mentioned make it so that that particular way of gaining root access is not possible, but it still allows an attacker to decrement and increment memory in specific locations so other exploits might be found to modify the system.

Now out-of-bound vulnerabilities are not new. Recently (february this year), a vulnerability in the networking code also provided an attack vector to get a local privilege escalation. A mandatory access control system like SELinux has little impact on such vulnerabilities if you allow users to execute their own code. Even confined users can modify the exploit to disable SELinux (since the shell code is ran with ring0 privileges it can access and modify the SELinux state information in the kernel).

Many thanks to Brad for the excellent write-up, and to the Gentoo Hardened team for providing the grSecurity PaX/TPE protections in its hardened-sources kernel.

We got back together on the #gentoo-hardened chat channel to discuss the progress of Gentoo Hardened, so it’s time for another write-up of what was said.

Toolchain

GCC 4.8.1 will be out soon, although nothing major has occurred with it since the last meeting. There is a plugin header install problem in 4.8 and its not certain that the (trivial) fix is in 4.8.1, but it certainly is inside Gentoo’s release.

Blueness is also (still, and hopefully for a long time ;-) maintaining the uclibc hardened related toolchain aspects.

Kernel and grSecurity/PaX

The further progress on the XATTR_PAX migration was put on a lower level the past few weeks due to busy, busy… very busy weeks (but this was announced and known in advance). We still need to do XATTR copying in install for packages that do pax markings before src_install() and include the user.pax XATTR patch in the gentoo-sources kernel. This will silence the errors for non-hardened users and fix the loss of XATTR markings for those packages that do pax-mark before install.

The set then needs to be documented further and tested on vanilla and hardened systems.

Zorry asked if a separate script can be provided for those ebuilds that directly call paxctl. These ebuilds might want to switch to the eclass, but if they need to call paxctl or similar directly (for instance because the result is immediately used for further building), a separate script or tool should be made available. Blueness will look into this.

On hardened-sources, we are now with stable 2.6.32-r160, 3.2.42-r1 and 3.8.6 due to some vulnerabilities in earlier versions (in networking code). There is still some bug (nfs-related) that is fixed in 3.2.44 so that part might need a bump as well soon.

SELinux

The selocal command is now available for Gentoo SELinux users, allowing them to easily enhance the policy without having to maintain their own SELinux policy modules (the script is a wrapper that does all that).

The setools package now also uses the SLOT’ed swig, so no more dependency breakage.

On SELinux userspace and policy, both have seen a new release last month, and both are already in the Gentoo portage tree.

Finally, the SELinux policy ebuilds now also call epatch_user so users can customize the policies even further without having to copy ebuilds to their overlay.

Now that tar supports XATTR well, we might want to look into SELinux stages again. Jmbsvicetto did some work on that, but the builds failed during stage1. We’ll look into that later.

Integrity

Nothing much to say, we’re waiting a bit until the patches proposed by the IMA team are merged in the main kernel.

Profiles

Two no-multilib fixes have been applied to the hardened/amd64/no-multilib profiles. One was a QA issue and quickly resolved, the other is due to the profile stacking within Gentoo profiles, where we missed a profile and thus were missing a few masks defined in that (missed) profile. But including the profile creates a lot of duplicates again, so we are going to copy the masks across until the duplicates are resolved in the other profiles.

Blueness will also clean up the experimental 13.0 directory since all hardened profiles now follow 13.0.

Docs

The latest changes on SELinux have been added to the Gentoo SELinux handbook. Also, I’ve been slowly (but surely) adding topics to the SELinux tutorials listing on the Gentoo wiki.

The grSecurity 2 document is very much out of date, blueness hopes to put some time in fixing that soon.

So that’s about it for the short write-up. Zorry will surely post the log later on the appropriate channels. Good work done (again) by all team members!

Extending SELinux policies with additional rules is easy. As SELinux uses a deny by default approach, all you need to do is to create a policy module that contains the additional (allow) rules, load that and you’re all set. But what if you want to remove some rules?

Well, sadly, SELinux does not support deny rules. Once an allow rule is loaded in memory, it cannot be overturned anymore. Yes, you can disable the module itself that provides the rules, but you cannot selectively disable rules. So what to do?

Generally, you can disable the module that contains the rules you want to disable, and load a custom module that defines everything the original module did, except for those rules you don’t like. For instance, if you do not want the skype_t domain to be able to read/write to the video device, create your own skype-providing module (myskype) with the exact same content (except for the module name at the first line) as the original skype module, except for the video device:

dev_read_sound(skype_t)
# dev_read_video_dev(skype_t)
dev_write_sound(skype_t)
# dev_write_video_dev(skype_t)

Load in this policy, and you now have the skype_t domain without the video access. You will get post-install failures when Gentoo pushes out an update to the policy though, since it will attempt to reload the skype.pp file (through the selinux-skype package) and fail because it declares types and attributes already provided (by myskype). You can exclude the package from being updated, which works as long as no packages depend on it. Or live with the post-install failure ;-) But there might be a simpler approach: epatch_user.

Recently, I added in support for epatch_user in the policy ebuilds. This allows users to create patches against the policy source code that we use and put them in /etc/portage/patches in the directory of the right category/package. For module patches, the working directory used is within the policy/modules directory of the policy checkout. For base, it is below the policy checkout (in other words, the patch will need to use the refpolicy/ directory base). But because of how epatch_user works, any patch taken from the base will work as it will start stripping directories up to the fourth one.

This approach is also needed if you want to exclude rules from interfaces rather than from the .te file: create a small patch and put it in /etc/portage/patches for the sec-policy/selinux-base package (as this provides the interfaces).

I use it since a long time, so since it works pretty good for me, I want to share how to handle the spam for your @gentoo.org address with procmail.

First, you need to say that procmail will filter your email(s):
echo "| /usr/bin/procmail" > /home/${USER}/.forward

Then create a simple /home/${USER}/.procmailrc with this content:
:0:
* ^X-Spam-Status: Yes
/dev/null

:0 B:
* ^Content-Type: .*; [ ]*(file)?name=\"?.*\.(pif|scr|com|cpl|vbs)\"?$
/dev/null

With the filter for X-Spam-Status and X-Spam-Level you will avoid the majority of the incoming spam.
Some mails that does not have any Spam flag, contains subject like viagra, cialis ( which I absolutely don’t need ), rolex and scount.
Yes, I could you the (c|C)ase syntax, but I had problems, so I prefer to write twice the rules instead of have any sort of troubles.
Note: with this email address I’m not subscribed to any newsletter or any sort of offers/catalogs so I filtered scount, markerting, money.

Sometimes I receive mails from people that are not spammer, with the X-Spam-Level flag with one star, so I decided to move these email into a folder, they will be double-checked with naked eye:

:0:
* ^X-Spam-Level: \*
/home/ago/.maildir/.INBOX.pspam/

To avoid confusion I always prefer to use a complete path here.

After a stabilization you will always see the annoying mail from the bugzilla which contains ${arch} stable, so if you want to drop them:

:0 B
* ^*(alpha|amd64|arm|hppa|ia64|m68k|ppc|ppc64|s390|sh|sparc|x86) stable*
/dev/null

Now, if you are using more email clients, on more computers, you may need to set the filters here instead of on all clients you are using, so for example:

:0
* ^From.*bugzilla-daemon@gentoo.org
* ^TO.*amd64@gentoo.org
/home/ago/.maildir/.INBOX.amd64/

And so on….
These, hints obviously are valid on all postfix-based mailserver; if you are using e.g. qmail, you need to move the .procmailrc, but this is still valid.
I hope this will help

EDIT:
If you need a particular set of rules, you can write it if you take a look at the source/header of the message, so If for example I don’t like to see the mails from bugzilla of the bugs that I reported:

the header says: X-Bugzilla-Reporter: ago@gentoo.org
so:

:0
* ^From.*bugzilla-daemon@gentoo.org
* ^X-Bugzilla-Reporter.*ago@gentoo.org
/dev/null

With all the reports surrounding Cdorked, I took a look at if SELinux and/or other Gentoo Hardened technologies could reduce the likelihood that this infection occurs on your system.

First of all, we don’t know yet how the malware gets installed on the server. We do know that the Apache binaries themselves are modified, so the first thing to look at is to see if this risk can be reduced. Of course, using an intrusion detection system like AIDE helps, but even with Gentoo’s qcheck command you can test the integrity of the files:

# qcheck www-servers/apache
Checking www-servers/apache-2.2.24 ...
  * 424 out of 424 files are good

If the binary is modified, this would result in something equivalent to:

Checking www-servers/apache-2.2.24 ...
 MD5-DIGEST: /usr/sbin/apache2
  * 423 out of 424 files are good

I don’t know if the modified binary would otherwise work just fine, I have not been able to find exact details on the infected binary to (in a sandbox environment of course) analyze this further. Also, because we don’t know how they are installed, it is not easy to know if binaries that you built yourself are equally likely to be modified/substituted or if the attack checks checksums of the binaries against a known list.

Assuming that it would run, then the infecting malware would need to set the proper SELinux context on the file (if it overwrites the existing binary, then the context is retained, otherwise it gets the default context of bin_t). If the context is wrong, then starting Apache results in:

apache2: Syntax error on line 61 of /etc/apache2/httpd.conf: Cannot load /usr/lib64/apache2/modules/mod_actions.so into server: /usr/lib64/apache2/modules/mod_actions.so: cannot open shared object file: Permission denied

This is because the modified binary stays in the calling domain context (initrc_t). If you use a targeted policy, then this will not present itself as initrc_t is an unconfined domain. But with strict policies, initrc_t is not allowed to read httpd_modules_t. Even worse, the remainder of SELinux protections don’t apply anymore, since with unconfined domains, all bets are off. That is why Gentoo focuses this hard on using a strict policy.

So, what if the binary runs in the proper domain? Well then, from the articles I read, the malware can do a reverse connect. That means that the domain will attempt to connect to an IP address provided by the attacker (in a specifically crafted URL). For SELinux, this means that the name_connect permission is checked:

# sesearch -s httpd_t -c tcp_socket -p name_connect -ACTS
Found 20 semantic av rules:
   allow nsswitch_domain dns_port_t : tcp_socket { name_connect } ; 
DT allow httpd_t port_type : tcp_socket { name_connect } ; [ httpd_can_network_connect ]
DT allow httpd_t ftp_port_t : tcp_socket { name_connect } ; [ httpd_can_network_relay ]
DT allow httpd_t smtp_port_t : tcp_socket { name_connect } ; [ httpd_can_sendmail ]
DT allow httpd_t postgresql_port_t : tcp_socket { name_connect } ; [ httpd_can_network_connect_db ]
DT allow httpd_t oracledb_port_t : tcp_socket { name_connect } ; [ httpd_can_network_connect_db ]
DT allow httpd_t squid_port_t : tcp_socket { name_connect } ; [ httpd_can_network_relay ]
DT allow httpd_t mssql_port_t : tcp_socket { name_connect } ; [ httpd_can_network_connect_db ]
DT allow httpd_t kerberos_port_t : tcp_socket { name_connect } ; [ allow_kerberos ]
DT allow nsswitch_domain ldap_port_t : tcp_socket { name_connect } ; [ authlogin_nsswitch_use_ldap ]
DT allow httpd_t http_cache_port_t : tcp_socket { name_connect } ; [ httpd_can_network_relay ]
DT allow httpd_t http_port_t : tcp_socket { name_connect } ; [ httpd_can_network_relay ]
DT allow httpd_t http_port_t : tcp_socket { name_connect } ; [ httpd_graceful_shutdown ]
DT allow httpd_t mysqld_port_t : tcp_socket { name_connect } ; [ httpd_can_network_connect_db ]
DT allow httpd_t ocsp_port_t : tcp_socket { name_connect } ; [ allow_kerberos ]
DT allow nsswitch_domain kerberos_port_t : tcp_socket { name_connect } ; [ allow_kerberos ]
DT allow httpd_t pop_port_t : tcp_socket { name_connect } ; [ httpd_can_sendmail ]
DT allow nsswitch_domain ocsp_port_t : tcp_socket { name_connect } ; [ allow_kerberos ]
DT allow httpd_t gds_db_port_t : tcp_socket { name_connect } ; [ httpd_can_network_connect_db ]
DT allow httpd_t gopher_port_t : tcp_socket { name_connect } ; [ httpd_can_network_relay ]

So by default, the Apache (httpd_t) domain is allowed to connect to DNS port (to resolve hostnames). All other name_connect calls depend on SELinux booleans (mentioned after it) that are by default disabled (at least on Gentoo). Disabling hostname resolving is not really feasible, so if the attacker uses a DNS port as port that the malware needs to connect to, SELinux will not deny it (unless you use additional networking constraints).

Now, the reverse connect is an interesting feature of the malware, but not the main one. The main focus of the malware is to redirect customers to particular sites that can trick the user in downloading additional (client) malware. Because this is done internally within Apache, SELinux cannot deal with this. As a user, make sure you configure your browser not to trust non-local iframes and such (always do this, not just because there is a possible threat right now). The configuration of Cdorked is a shared memory segment of Apache itself. Of course, since Apache uses shared memory, the malware embedded within will also have access to the shared memory. However, if this shared memory would need to be accessed by third party applications (the malware seems to grant read/write rights on everybody to this segment) SELinux will prevent this:

# sesearch -t httpd_t -c shm -ACTS
Found 2 semantic av rules:
   allow unconfined_domain_type domain : shm { create destroy getattr setattr read write associate unix_read unix_write lock } ; 
   allow httpd_t httpd_t : shm { create destroy getattr setattr read write associate unix_read unix_write lock } ; 

Only unconfined domains and the httpd_t domain itself have access to httpd_t labeled shared memory.

So what about IMA/EVM? Well, those will not help here since IMA checks for integrity of files that were modified offline. As the modification of the Apache binaries is most likely done online, IMA would just accept this.

For now, it seems that a good system integrity approach is the most effective until we know more about how the malware-infected binary is written to the system in the first place (as this is better protected by MAC controls like SELinux).

When working on (the still on-going) migration of the Gentoo java project repositories from SVN to Git I ran into bugs with svn2git 1.0.8 and my own svneverever 1.2.1.

The bug with svn2git 1.0.8 was a regression that broke support for (non-ASCII) UTF-8 author names in identity maps. That’s fixed in dev-vcs/svn2git-1.0.8-r1 in Gentoo. I sent the patch upstream and to the Debian package maintainer, too.

For svneverever, a directory that re-appeared after deletion was reported to only live once, e.g. the output was

(2488; 9253)  /projects
(2490; 9253)      /java-config-2
(2490; 2586)          /trunk

if directory /projects/java-config-2/trunk/ got deleted at revision 2586, no matter if was re-created later. With 9253 revisions in total, the correct output (with svneverever 1.2.2) is:

(2488; 9253)  /projects
(2490; 9253)      /java-config-2
(2490; 9253)          /trunk

That’s fixed in svneverever 1.2.2.

If svneverever is of help to you, please support me on Flattr. Thanks!

Recently, the metadata.xml file syntax definition (the DTD for those that know a bit of XML) has been updated to support CPE definitions. A CPE (Common Platform Enumeration) is an identifier that describes an application, operating system or hardware device using its vendor, product name, version, update, edition and language. This CPE information is used in the CVE releases (Common Vulnerabilities and Exposures) – announcements about vulnerabilities in applications, operating systems or hardware. Not all security vulnerabilities are assigned a CVE number, but this is as close as you get towards a (public) elaborate dictionary of vulnerabilities.

By allowing Gentoo package maintainers to enter (part of) the CPE information in the metadata.xml file, applications that parse the CVE information can now more easily match if software installed on Gentoo is related to a CVE. I had a related post to this not that long ago on my blog and I’m glad this change has been made. With this information at hand, we can start feeding CPE information to the packages and then easily match this with CVEs.

I had a request to “provide” the scripts I used for the previous post. Mind you, these are taking too many assumptions (and probably wrong ones) for now (and I’m not really planning on updating them as I have different methods for getting information related to CVEs), but I’m planning on integrating CPE data in Gentoo’s packages more and then create a small script that generates a “watchlist” that I can feed to cvechecker. But anyway, here are the scripts.

First, I took all CVE information and put it in a simple CSV file. The CSV is the same one used by cvechecker, so check out the application to see where it fetches the data from (there is a CVE RSS feed and a simple XSL transformation). Second, I create a “hitlist” which generates the CPEs. With the recent change to metadata.xml this step can be simplified a lot. Third, I try to match the CPE data with the CVE data, depending on a given time delay of commits. In other words, you can ask possible CVE fixes for commits made in the last few XXX days.

Being long overdue – like many of our documentation-reported bugs :-( I worked on bug 466262 to update the Gentoo Handbook with information about Network Interface Naming. Of course, the installation instructions have also seen the necessary updates to refer to this change.

With some luck (read: time) I might be able to fix various other documentation-related ones soon. I had some problems with the new SELinux userspace that I wanted to get fixed before, and then I worked on the new SELinux policies as well as trying to figure out how SELinux deals with network related aspects. Hence I saw time fly by at the speed of a neutrino…

BTW, the 20130424 policies are in the tree.

Since a long time I realized that is a pita every time that I keyword, receive a repoman failure for dependency.bad(mostly) that does not regard the arch that I’m changing.
So, checking in the repoman manual, I realized that –ignore-arches looks bad for my case and I decided to request a new feature: –include-arches.
This feature, as explained in the bug, checks only for the arches that you write as argument and should be used only when you are keywording/stabilizing.

Some examples/usage:

First, it saves time, the following example will try to run repoman full in the kdelibs directory:
$ time repoman full > /dev/null 2>&1
real 0m12.434s

$ time repoman full --include-arches "amd64" > /dev/null 2>&1
real 0m3.880s

Second, kdelibs suffers for a dependency.bad on amd64-fbsd, so:
$ repoman full
RepoMan scours the neighborhood...
>>> Creating Manifest for /home/ago/gentoo-x86/kde-base/kdelibs
dependency.bad 2
kde-base/kdelibs/kdelibs-4.10.2.ebuild: PDEPEND: ~amd64-fbsd(default/bsd/fbsd/amd64/9.0) ['>=kde-base/nepomuk-widgets-4.10.2:4[aqua=]']

$ repoman full --include-arches "amd64"
RepoMan scours the neighborhood...
>>> Creating Manifest for /home/ago/gentoo-x86/kde-base/kdelibs

Now when I will keyword the packages I can check for specific arches and skip the unuseful checks since they causes, in this case, only a waste of time.
Thanks to Zac for the work on it.

Learn more at dayagainstdrm.org (and drm.info).

If you’re a university student, time is running out! You could get paid to hack on Gentoo or other open-source software this summer, but you’ve gotta act now. The deadline to apply for the Google Summer of Code is this Friday.

If this sounds like your dream come true, you can find some Gentoo project ideas here and Gentoo’s GSoC homepage here. For non-Gentoo projects, you can scan through the GSoC website to find the details.

When you read through the Gentoo Hardened SELinux handbook, you’ll notice that we sometimes update /etc/fstab with some SELinux-specific settings. So, what are these settings about and are there more of them?

First of all, let’s look at a particular example from the installation instructions so you see what I am talking about:

tmpfs  /tmp  tmpfs  defaults,noexec,nosuid,rootcontext=system_u:object_r:tmp_t  0 0

What the rootcontext= option does here is to set the context of the “root” of that file system (meaning, the context of /tmp in the example) to the specified context before the file system is made visible to the userspace. Because we do it soon, the file system is known as tmp_t throughout its life cycle (not just after the mount or so).

Another option that you’ll frequently see on the Internet is the context= option. This option is most frequently used for file systems that do not support extended attributes, and as such cannot store the context of files on the file system. With the context= mount option set, all files on that file system get the specified context. For instance, context=system_u:object_r:removable_t.

If the file system does support extended attributes, you might find some benefit in using the defcontext= option. When set, the context of files and directories (and other resources on that file system) that do not have a SELinux context set yet will use this default context. However, once a context is set, it will use that context instead.

The last context-related mount option is fscontext=. With this option, you set the context of the “filesystem” class object of the file system rather than the mount itself (or the files). Within SELinux, “filesystem” is one of the resource classes that can get a context. Remember the /tmp mount example from before? Well, even though the files are labeled tmp_t, the file system context itself is still tmpfs_t.

It is important to know that, if you use one of these mount options, context= is mutually exclusive to the other options as it “forces” the context on all resources (including the filesystem class).

Hi all, long time no see.

Some weeks ago I got an early version of the BeagleBone Black from the people at Beagleboard.org to create the documentation I always create with every device I get.

Like always i’d like to announce the guide for installing Gentoo in the BeagleBone Black. Have a look at: http://dev.gentoo.org/~armin76/arm/beagleboneblack/install.xml . Feel free to send any corrections my way.

This board is a new version of the original BeagleBone, known in the community as BeagleBone white, for which I wrote a post for it: http://armin762.wordpress.com/2012/01/01/beaglebone-and-gentoo/

This new version differs in some aspects with the previous version:

• Cheaper: 45$ vs 89$ of the BeagleBone white
• 512MB DDR3L RAM vs 256MB DDR2 RAM of the BeagleBone white
• 1GHz of processor speed vs 720MHz of the BeagleBone white, both when using an external PSU for power

Also it has more features which the old BeagleBone didn’t had

• miniHDMI output
• 2GB eMMC

However the new version has missing:

• Serial port and JTAG through the miniUSB interface

The reason for missing this feature is cost cutting measures, as can be read in the Reference manual.

The full specs of the BeagleBone Black are:
# ARMv7-A 1GHz TI AM3358/9 ARM Cortex-A8 processor
# 512MB DDR3L RAM
# SMSC LAN8710 Ethernet card
#
# 1x microSDHC slot
# 1x USB 2.0 Type-A port
# 1x mini-USB 2.0 OTG port
# 1x RJ45
# 1x 6 pin 3.3V TTL Header for serial
#
# Reset, power and user-defined button

More info about the specs in BeagleBone Black’s webpage.

For those curious as me, here’s the bootlog and the cpuinfo.

I’ve found two issues while working on it:

1. The USB port doesn’t have a working hotplug detection. That means that if you plug an USB device in the USB port, it will be only detected once, if you remove the USB device, the USB port will stop working. I’ve been told that they are working on it. I haven’t been able to find a workaround for it.
2. The BeagleBone Black doesn’t detect an microSD card when plugged in when its been booted from the eMMC. If you want to use a microSD card for additional storage, it must be inserted before it boots.

I’d like to thank the people at Beagleboard.org for providing me a Beaglebone Black to document this.

Have fun!

I’ve recently came across libffi again. No, not because it was mentioned during the Gentoo Hardened online meeting, but because my /var/tmp wasn’t mounted correctly, and emerge (actually python) uses libffi. Most users won’t notice this, because libffi works behind the scenes. But when it fails, it fails bad. And SELinux actually helped me quickly identify what the problem is.

$ emerge --info
segmentation fault

The abbreviation “libffi” comes from Foreign Function Interface, and is a library that allows developers to dynamically call code from another application or library. But the method how it approaches this concerns me a bit. Let’s look at some strace output:

8560  open("/var/tmp/ffiZ8gKPd", O_RDWR|O_CREAT|O_EXCL, 0600) = 11
8560  unlink("/var/tmp/ffiZ8gKPd")      = 0
8560  ftruncate(11, 4096)               = 0
8560  mmap(NULL, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 11, 0) = -1 EACCES (Permission denied)

Generally, what libffi does, is to create a file somewhere where it can write files (it checks the various mounts on a system to get a list of possible target file systems), adds the necessary data (that it wants to execute) to it, unlinks the file from the file system (but keep the file descriptor open, so that the file cannot (easily) be modified on the system anymore) and then maps it to memory for executable access. If executing is allowed by the system (for instance because the mount point does not have noexec), then SELinux will trap it because the domain (in our case now, portage_t) is trying to execute an (unlinked) file for which it holds no execute rights on:

type=AVC msg=audit(1366656205.201:2221): avc:  denied  { execute } for  
pid=8560 comm="emerge" path=2F7661722F66666962713154465A202864656C6574656429 
dev="dm-3" ino=6912 scontext=staff_u:sysadm_r:portage_t tcontext=staff_u:object_r:var_t
tclass=file

When you notice something like this (an execute on an unnamed file), then this is because the file descriptor points to a file already unlinked from the system. Finding out what it was about might be hard (but with strace it is easy as … well, whatever is easy for you).

Now what happened was that, because /var/tmp wasn’t mounted, files created inside it got the standard type (var_t) which the Portage domain isn’t allowed to execute. It is allowed to execute a lot of types, but not that one ;-) When /var/tmp is properly mounted, the file gets the portage_tmp_t type where it does hold execute rights for.

Now generally, I don’t like having world-writeable locations without noexec. For /tmp, noexec is enabled, but for /var/tmp I have (well, had ;-) to allow execution from the file system, mainly because some (many?) Gentoo package builds require it. So how about this dual requirement, of allowing Portage to write (and execute) its own files, and allow libffi to do its magic? Certainly, from a security point of view, I might want to restrict this further…

Well, we need to make sure that the location where Portage works with (the location pointed to by $PORTAGE_TMPDIR) is specifically made available for Portage: have the directory only writable by the Portage user. I keep it labeled as tmp_t so that the existing policies apply, but it might work with portage_tmp_t immediately set as well. Perhaps I’ll try that one later. With that set, we can have this mount-point set with exec rights (so that libffi can place its file there) in a somewhat more secure manner than allowing exec on world-writeable locations.

So now my /tmp and /var/tmp (and /run and /dev/shm and /lib64/rc/init.d) are tmpfs-mounts with the noexec (as well as nodev and nosuid) bits set, with the location pointed towards by $PORTAGE_TMPDIR being only really usable by the Portage user:

$ ls -ldZ /var/portage
drwxr-x---. 4 portage root system_u:object_r:tmp_t 4096 Apr 22 21:45 /var/portage/

And libffi? Well, allowing applications to create their own executables and executing it is something that should be carefully governed. I’m not aware of any existing or past vulnerabilities, but I can imagine that opening the ffi* file(s) the moment they come up (to make sure you have a file descriptor) allows you to overwrite the content after libffi has created it but before the application actually executes it. By limiting the locations where applications can write files to (important step one) and the types they can execute (important step two) we can already manage this a bit more. Using regular DAC, this is quite difficult to achieve, but with SELinux, we can actually control this a bit more.

Let’s first see how many domains are allowed to create, write and execute files:

$ sesearch -c file -p write,create,execute -A | grep write | grep create \
  | grep execute | awk '{print $1}' | sort | uniq | wc -l
32

Okay, 32 target domains. Not that bad, and certainly doable to verify manually (hell, even in a scripted manner). You can now check which of those domains have rights to execute generic binaries (bin_t), possibly needed for command execution vulnerabilities or privilege escalation. Or that have specific capabilities. And if you want to know which of those domains use libffi, you can use revdep-rebuild to find out which files are linked to the libffi libraries.

It goes to show that trying to keep your box secure is a never-ending story (please, companies, allow your system administrators to do their job by giving them the ability to continuously increase security rather than have them ask for budget to investigate potential security mitigation directives based on the paradigm of business case and return on investment using pareto-analytics blaaaahhhh….), and that SELinux can certainly be an important method to help achieve it.

mongoDB 2.4.3

Yet another bugfix release, this new stable branch is surely one of the most quickly iterated I’ve ever seen. I guess we’ll wait a bit longer at work before migrating to 2.4.x.

pacemaker 1.1.10_rc1

This is the release of pacemaker we’ve been waiting for, fixing among other things, the ACL problem which was introduced in 1.1.9. Andrew and others are working hard to get a proper 1.1.10 out soon, thanks guys.

Meanwhile, we (gentoo cluster herd) have been contacted by @Psi-Jack who has offered his help to follow and keep some of our precious clustering packages up to date, I wish our work together will benefit everyone !

All of this is live on portage, enjoy.

A new release of the SELinux userspace utilities was recently announced. I have made the packages for Gentoo available and they should now be in the main tree (~arch of course). During the testing of the packages however, I made a stupid mistake of running the tests on the wrong VM, one that didn’t contain the new packages. Result: no regressions (of course). My fault for not using in-ebuild tests properly, as I should. So you’ll probably see me blogging about the in-ebuild testing soon ;-)

In any case, the regressions I did find out (quite fast after I updated my main laptop with them as well) where a missing function in libselinux, a referral to a non-existing makefile when using “semanage permissive” and the new sepolicy application requiring yum python bindings. At least, with the missing function (hopefully correctly) resolved, all tests I usually do (except for the permissive domains) are now running well again.

This only goes to show how important testing is. Of course, I reported the bugs on the mailinglist of the userspace utilities as well. Hopefully they can look at them while I’m asleep so I can integrate fixes tomorrow more easily ;-)

Hey Gentoo folks !

I finally followed a friend’s advice and stepped into the Gentoo Planet and Universe feeds. I hope my modest contributions will help and be of interest to some of you readers.

As you’ll see, I don’t talk only about Gentoo but also about photography and technology more generally. I also often post about the packages I maintain or I have an interest in to highlight their key features or bug fixes.

• kleopatra-4.4 segfaults immediately on start
• kmail-4.4 does not respect "Sent mail" folder of identity 
• session restore always complains that another kontact-4.4 instance is running
• on each login, a dialog box about (unfounded) problems with gnupg pops up

• end of april 2013, all kdepim-4.4 packages in the Gentoo portage tree will be package.masked 
• end of may 2013, all kdepim-4.4 packages in the Gentoo portage tree will be removed
• afterwards, we will finally be able to simplify the eclasses a lot by removing the special handling

After the security issue related bumps of the previous releases which happened last weeks it was about time 10gen released a 2.4.x fixing the following issues:

• Fix for upgrading sharded clusters
• TTL assertion on replica set secondaries
• Several V8 memory leak and performance fixes
• High volume connection crash

I guess everything listed above would have affected our cluster at work so I’m glad we’ve been patient on following-up this release See the changelog for details.

The smart cards

The readers

Drivers and software

gpg --card-status

ACTION=="add", SUBSYSTEM=="usb", ENV{PRODUCT}=="4e6/e003/*", OWNER:="huettel", MODE:="600"
ACTION=="add", SUBSYSTEM=="usb", ENV{PRODUCT}=="4e6/5115/*", OWNER:="huettel", MODE:="600"

udevadm control --reload-rules

killall -9 scdaemon

huettel@pinacolada ~ $ gpg --card-status
Application ID ...: D276000124010200000500000AFA0000
Version ..........: 2.0
Manufacturer .....: ZeitControl
Serial number ....: 00000AFA
Name of cardholder: [not set]
Language prefs ...: de
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: 2048R 2048R 2048R
Max. PIN lengths .: 32 32 32
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
huettel@pinacolada ~ $

Here’s a response to an email thread I sent recently. This was on a private alias but I’m not exposing the context or quoting anybody, so I’m not leaking anything but my own opinion which has no reason to be secret.

GLEP39 explicitly states that projects can be competing. I don’t see how you can exclude competing ebuilds from that since nothing prevents anybody from starting a project dedicated to maintaining an ebuild.

So, if you want to prevent devs from pushing competing ebuilds to the tree you have to change GLEP 39 first. No arguing or “hey all, hear my opinion” emails on whatever list will be able to change that.

Some are against forking ebuilds and object duplicating effort and lack of manpower. I will bluntly declare those people shortsighted. Territoriality is exactly what prevents us from getting more manpower. I’m interested in improving package X but developer A who maintains it is an ass and won’t yield on anything. At best I’ll just fork it in an overlay (with all the issues that having a package in an overlay entail, i.e. no QA, it’ll die pretty quickly, etc…), at worst I’m moving to Arch, or Exherbo, or else… What have we gained by not duplicating effort? We have gained negative manpower.

As long as forked ebuilds can cohabit peacefully in the tree using say a virtual (note: not talking about the devs here but about the packages) we should see them as progress. Gentoo is about choice. Let consumers, i.e. users and devs depending on the ebuild in various ways, have that choice. They’ll quickly make it known which one is best, at which point the failing ebuild will just die by itself. Let me say it again: Gentoo is about choice.

If it ever happened that devs of forked ebuilds could not cohabit peacefully on our lists or channels, then I would consider that a deliberate intention of not cooperating. As with any deliberate transgression of our rules if I were devrel lead right now I would simply retire all involved developers on the spot without warning. Note the use of the word “deliberate” here. It is important we allow devs to make mistakes, even encourage it. But we are adults. If one of us knowingly chooses to not play by the rules he or she should not be allowed to play. “Do not be an ass” is one of those rules. We’ve been there before with great success and it looks like we are going to have to go there again soon.

There you have it. You can start sending me your hate mail in 3… 2… 1…

Just wanted to let everybody know that I’m in Ireland, as I landed at Dublin Airport on Saturday, and been roaming around the city for a few days now. Time looks like it’s running faster than usual, so I haven’t had much time to work on Gentoo stuff.

My current plan is to work, by the end of the week, on a testing VM as there’s an LVM2 bug that I owe Enrico to fix, and possibly work on the Autotools Mythbuster guide as well, there’s work to do there.

But today, I’m a bit too tired to keep going, it’s 11pm… I’ll doze off!

While Spring has yet to come here, the rainy days are giving me some time to think about the future of Sabayon and summarize what’s been done during the last months.

donations

As far as I can see, donations are going surprisingly well. The foundation has now enough money (see the pledgie.com campaign at sabayon.org) to guarantee 24/7 operations, new hardware purchase and travel expenses for several months. Of course, the more the better (paranoia mode on) but I cannot really complain, given that’s our sole source of funds. Here is a list of stuff we’ve been able to buy during the last year (including prices, we’re in the EU, prices in the US are much lower, sigh):

• one Odroid X2 (for Sabayon on ARM experiments) – 131€
• one PandaBoard ES (for Sabayon on ARM experiments) – 160€
• two 2TB Seagate Barracuda HDDs (one for Joost’s experiments, one for the Entropy tinderbox) – 185€
• two 480GB Vertex3 OCZ SSDs for the Entropy tinderbox (running together with the Samsung 830 SSDs in a LVM setup) – 900€
• one Asus PIKE 2008 SAS controller for the Entropy tinderbox – 300€
• other 16GB of DDR3 for the Entropy tinderbox (now running with 64G) – 128€
• mirror.de.sabayon.org @ hetzner.de maintenance (33€/mo for 1 year) – 396€
• my personal FOSDEM 2013 travel expenses – 155€

Plus, travel expenses to data centers whenever there is a problem that cannot be fixed remotely. That’s more or less from 40€ to 60€ each depending on the physical distance.
As you may understand, this is just a part of the “costs”, because the time donated by individual developers is not accounted there, and I believe that it’s much more important than a piece of silicon.

monthly releases, entropy

Besides the money part, I spent the past months on Sabayon 11 (of course), on advancing with the automation agenda for 2013. Ideally, I would like to have stable releases automatically produced and tested monthly, and eventually pushed to mirrors. This required me to migrate to a different bittorrent tracker, one that scrapes a directory containing .torrents and publishes them automatically: you can see the outcome at http://torrents.sabayon.org. Furthermore, a first, yet not advertised, set of monthly ISO images is available on our mirrors into the iso/monthly/ sub-directory. You can read more about them here. This may (eheh) indicate that the next Sabayon release will be versioned something like 13.05, who knows…
On the Entropy camp, nothing much has changed, besides the usual set of bug fixe, little improvements and the migration to an .ini-like repositories configuration files syntax for both Entropy Server and Client modules, see here. You may start realizing that all the good things I do are communicated through the devel mailing list.

leh systemd

I spent a week working on a Sabayon systemd system to see how it works and performs compared to openrc. Long story short, I am about to arrange some ideas on making the systemd migration come true at some point in the (near) future. Joost and I are experimenting with a private Entropy repository (thus chroot) that’s been migrated to systemd, from openrc. While I don’t want to start yet another flamewar about openrc vs systemd, I do believe in science, facts and benchmarks. Even though I don’t really like the vertical architecture of systemd, I am starting to appreciate its features and most importantly, its performance. The first thing I would like to sort out is to be able to switch between systemd and openrc at runtime, this may involve the creation of an eselect module (trivial) and patching some ebuilds. I think that’s the best thing to do, if we really want to design and deploy a migration path for current openrc users (I would like to remind people that Gentoo is about choice, after all). If you’re a Gentoo developer that hasn’t been bugged by me yet, feel free to drop a line to lxnay@g.o (expand the domain, duh!) if you’re interested.

FOSDEM 2013

I started writing this post after FOSDEM, but never actually managed to finish it. But as I plan to blog about something again “soon”, I wanted to get this one out first. So let’s start with FOSDEM – it is awesome event and every open source hacker is there unless he has some really huge reasons why not to come (like being dead, in prison or locked down in psychiatric care). I was there together with bunch of openSUSE/SUSE folks. It was a lot of fun and we even managed to get some work done during the event. So how was it?

FOSDEM

We had a lot of fun on the way already. You know, every year, we rent a bus just for us and we go from Nuremberg to Brussels and back all together by bus. And we talk and drink and talk and drink some more…. So although it sounds crazy – 8 hours drive – it’s not as bad as it sounds.

etc-update

What the hack is etc-update and what does it have to do with me, openSUSE or FOSDEM? Isn’t it Gentoo tool? Yes, it is. It is Gentoo tool (actually part of portage, Gentoo package manager) that is used in Gentoo to merge updates to the configuration files. When you install package, portage is not going to overwrite your configuration files that you have spend days and nights tuning. It will create a new file with new upstream configuration and it is up to you to merge them. But you know, rpm does the same thing. In almost all cases rpm is not going to overwrite your configuration file, but will install the new one as config_file.rpmnew. And it is up to you to merge the changes. But it’s not fun. Searching for all files, compare them manually and choose what to merge and how. And here comes etc-update o the rescue

How does it work? Simple. You need to install it (will speak about that later) and run it. It’s command line tool and it doesn’t need any special parameters. All you need to do is to run etc-update as root (to be actually able to do something with these files). And the result?

# etc-update 
Scanning Configuration files...
The following is the list of files which need updating, each
configuration file is followed by a list of possible replacement files.
1) /etc/camsource.conf (1)
2) /etc/ntp.conf (1)
Please select a file to edit by entering the corresponding number.
              (don't use -3, -5, -7 or -9 if you're unsure what to do)
              (-1 to exit) (-3 to auto merge all files)
                           (-5 to auto-merge AND not use 'mv -i')
                           (-7 to discard all updates)
                           (-9 to discard all updates AND not use 'rm -i'):

What I usually do is that I select configuration files I do care about, review changes and merge them somehow and later just use -5 for everything else. It looks really simple, doesn’t it? And in fact it is!

Somebody asked a question on how to merge updates of configuration files while visiting our openSUSE booth at FOSDEM. When I learned that from Richard, we talked a little bit about how easy it is to do something like that and later during one of the less interesting talks, I took this Gentoo tool, patched it to work on rpm distributions, packaged it and now it is in Factory and it will be part of openSUSE 13.1 If you want to try it, you can get it either from my home project – home:-miska-:arm (even for x86 ) or from utilities repository.

Hope you will like it and that it will make many sysadmins happy

If you’re using dev-db/postgresql-server, update now.

CVE-2013-1899 <dev-db/postgresql-server-{9.2.4,9.1.9,9.0.13}
------------------------------------------------------------
A connection request containing a database name that begins
with "-" may be crafted to damage or destroy files within a server's data directory.

CVE-2013-1900 <dev-db/postgresql-server-{9.2.4,9.1.9,9.0.13,8.4.17}
-------------------------------------------------------------------
Random numbers generated by contrib/pgcrypto functions may be easy for another
database user to guess

CVE-2013-1901 <dev-db/postgresql-server-{9.2.4,9.1.9}
-----------------------------------------------------
An unprivileged user can run commands that could interfere with in-progress backups.

The article’s title is a play on the phrase “don’t open that door”, and makes more sense in Italian as we use the same word for ‘door’ and ‘port’…

So you left your hero (me) working on setting up a Raspberry Pi with at least a partial base of cross-compilation. The whole thing worked to a decent extent, but it wasn’t really as feasible as I hoped. Too many things, including Python, cannot cross-compile without further tricks, and the time it takes to figure out how to cross-compile them, tend to be more than that needed to just wait for it to build on the board itself. I guess this is why there is that little interest in getting cross-compilation supported.

But after getting a decent root, or stage4 as you prefer to call it, I needed to get a kernel to boot the device. This wasn’t easy.; there is no official configuration file published — what they tell you is, if you want to build a new custom kernel, to zcat /proc/config.gz from Raspian. I didn’t want to use Raspian, so I looked further. The next step is to check out the defconfig settings that the kernel repository includes, a few, different of them exist.

You’d expect them to be actually thought out to enable exactly what the RaspberryPi provides, and nothing more or less. Some leeway can be expected for things like network options, but at least the “cutdown” version should not include all of IrDA, Amateur Radio, Wireless, Bluetooth, USB network, PPP, … After disabling a bunch of options, since the system I need to run will have very few devices connected – in particular, only the Davis Vantage Pro station, maybe a printer – I built the kernel and copied it over the SD card. It booted, it crashed. Kernel panicked right away, due to a pointer dereference.

After some rebuild-copy-test cycles I was able to find out what the problem is. It’s a problem that is not unique to the RPi actually, as I found the same trace from an OMAP3 user reporting it somewhere else. The trick was disabling the (default-enabled) in-kernel debugger – which I couldn’t access anyway, as I don’t have an USB keyboard at hand right now – so that it would print the full trace of the error .That pointed at the l4_init function, which is the initialization of the Lightning 4 gameport controller — an old style, MIDI game port.

My hunch is that this expansion card is an old-style ISA card, since it does not rely on PCI structures to probe for the device — I cannot confirm it because googling for “lightning 4” only comes up with images of iPad and accessories. What it does, is simply poking at the 0×201 address, and the moment when it does, you get a bad dereference from the kernel exactly at that address. I’ve sent a (broken, unfortunately) patch to the LKML to see if there is an easy way to solve this.

To be honest and clear, if you just take a defconfig and build it exactly as-is, you won’t be hitting that problem. The problem happens to me because in this kernel, like in almost every other one I built, I do one particular thing: I disable modules so that a single, statically build kernel. This in turn means that all the drivers are initialized when you start the kernel, and the moment when the L4 driver is started, it crashes the kernel. Possibly it’s not the only one.

This is most likely not strictly limited to the RaspberryPi but it doesn’t help that there is no working minimal configuration – mine is, by the way, available here – and I’m pretty sure there are other similar situations even when the arch is x86… I guess it’s just a matter of reporting them when you encounter them.

This is part two of a series of articles looking into the new udev “predictable” names. Part one is here and talks about the path-based names.

As Steve also asked on the comments from last post, isn’t it possible to just use the MAC address of an interface to point at it? Sure it’s possible! You just need to enable the mac-based name generator. But what does that mean? It means that your new interface names will be enx0026b9d7bf1f and wlx0023148f1cc8 — do you see yourself typing them?

Myself, I’m not going to type them. My favourite suggestion to solve the issue is to rely on rules similar to the previous persistent naming, but not re-using the eth prefix to avoid collisions (which will no longer be resolved by future versions of udev). I instead use the names wan0 and lan0 (and so on), when the two interfaces sit stranding between a private and a public network. How do I achieve that? Simple:

SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:17:31:c6:4a:ca", NAME="lan0"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:07:e9:12:07:36", NAME="wan0"

Yes these simple rules are doing all the work you need if you just want to make sure not to mix the two interfaces by mistake. If your server or vserver only has one interface, and you want to have it as wan0 no matter what its mac address is (easier to clone, for instance), then you can go for

SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="*", NAME="wan0"

As long as you only have a single network interface, that will work just fine. For those who use Puppet, I also published a module that you can use to create the file, and ensure that the other methods to achieve “sticky” names are not present.

My reasoning to actually using this kind of names is relatively simple: the rare places where I do need to specify the interface name are usually in ACLs, the firewall, and so on. In these, the most important part to me is knowing whether the interface is public or not, so the wan/lan distinction is the most useful. I don’t intend trying to remember whether enp5s24k1f345totheright4nextothebaker is the public or private interface.

Speaking about which, one of the things that appears obvious even from Lennart’s comment to the previous post, is that there is no real assurance that the names are set in stone — he says that an udev upgrade won’t change them, but I guess most people would be sceptic, remembering the track record that udev and systemd has had over the past few months alone. In this situation my personal, informed opinion is that all this work on “predictable” names is a huge waste of time for almost everybody.

If you do care about stable interface names, you most definitely expect them to be more meaningful than 10-digits strings of paths or mac addresses, so you almost certainly want to go through with custom naming, so that at least you attach some sense into the names themselves.

On the other hand, if you do not care about interface names themselves, for instance because instead of running commands or scripts, you just use NetworkManager… well what the heck are you doing playing around with paths? If it doesn’t bother you that the interface for an USB device changes considerably between one port and another, how can it matter to you whether it’s called wwan0 or wwan123? And if the name of the interface does not matter to you, why are you spending useless time trying to get these “predictable” names working?

All in all, I think this is just an useless nice trick, that will only cause more headaches than it can possibly solve. Bahumbug!

This is going to be fun. The Gentoo “udev team”, in the person of Samuli – who seems to suffer from 0-day bump syndrome – decided to now enable by default the new predictable names featuer that is supposed to make things so much nicer in Linux land where, especially for people coming from FreeBSD, things have been pretty much messed up. This replaces the old “persistent” names, that were often enough too fragile to work, as they did in-place renaming of interfaces, and would cause way too often conflicts at boot time, since swapping two devices’ names is not an atomic operation for obvious reasons.

So what’s this predictable name all around? Well, it’s mostly a merge of the previous persistent naming system, and the BIOS label naming project which was developed by RedHat for a few years already so that the names of interfaces for server hardware in the operating system match the documentation of said server, so that you can be sure that if you’re connecting the port marked with “1” on the chassis, out of four on the motherboard, it will bring up eth2.

But why were those two technologies needed? Let’s start first with explaining how (more or less) the kernel naming scheme works: unlike the BSD systems, where the interfaces are named after the kernel driver (en0, dc0, etc.), the Linux kernel uses generic names, mostly eth, wlan and wwan, and maybe a couple more, for tunnels and so on. This causes the first problem: if you have multiple devices of the same class (ethernet, wlan, wwan) coming from different drivers, the order of the interface may very well vary between reboots, either because of changes in the kernel, if the drivers are built-in, or simply because of locking and execution of modules load (which is much more common for binary distributions).

The reason why changes in the kernel can change the order is that the order in which drivers are initialized has changed before and might change again in the future. A driver could also decide to change the order with which its devices are initialized (PCI tree scanning order, PCI ID order, MAC address order, …) and so on, causing it to change the order of interfaces even for the same driver. More about this later.

But here’s my first doubt arises: how common is for people to have more than one interface of the same class from vendors different enough to use different drivers? Well it depends on the class of device; on a laptop you’d have to search hard for a model with more than one Ethernet or wireless interface, unless you add an ExpressCard or PCMCIA expansion card (and even those are not that common). On a desktop, I’ve seen a few very recent motherboards with more than one network port, and I have yet to see one with different chips for the two. Servers, that’s a different story.

Indeed, it’s not that uncommon to have multiple on-board and expansion card ports on a server. For instance you could use the two onboard ports as public and private interfaces for the host… and then add a 4-port card to split between virtual machines. In this situation, having a persistent naming of the interfaces is indeed something you would be glad of. How can you tell which one of eth{0..5} is your onboard port #2, otherwise? This would be problem number two.

Another situation in which having a persistent naming of interfaces is almost a requirement is if you’re setting up a router: you definitely don’t want to switch the LAN and WAN interface names around, especially where the firewall is involved.

This background is why the persistent-net rules were devised quite a few years ago for udev. Unfortunately almost everybody got at least one nasty experience with them. Sometimes the in-place rename would fail, and you’d end up with the temporary names at the end of boot. In a few cases the name was not persistent at all: if the kernel driver for the device would change, or change name at least, the rules wouldn’t match and your eth0 would become eth1 (this was the case when Intel split the e1000 and e1000e drivers, but it’s definitely more common with wireless drivers, especially if they move from staging to main).

So the old persistent net rules were flawed. What about the new predictable rules? Well, not only they combined the BIOS naming scheme (which is actually awesome when it works — SuperMicro servers such as Excelsior do not expose the label; my Dell laptop only exposes a label for the Ethernet port but doesn’t for either the wireless adapter or the 3G one), but it has two “fallbacks” that are supposed to be used when the labels fail, one based on the MAC address of the interface, and the other based on the “path” — which for most PCI, PCI-E, onboard, ExpressCard ports is basically the PCI address; for USB… we’ll see in a moment.

So let’s see, from my laptop:

# lspci | grep &aposNetwork controller&apos
03:00.0 Network controller: Intel Corporation Centrino Advanced-N 6200 (rev 35)
# ifconfig | grep wlp3
wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

Why “wlp3s0”? It’s the Wireless adapter (wl) PCI (p) card at bus 3, slot 0 (s0): 03:00.0. Matches lspci properly. But let’s see the WWAN interface on the same laptop:

# ifconfig -a | grep ww
wwp0s29u1u6i6: flags=4098<BROADCAST,MULTICAST>  mtu 1500

Much longer name! What’s going on then? Let’s see, it’s reporting it’s card at bus 0, slot 29 (0×1d) — lspci will use hexadecimal numbers for the addresses:

# lspci | grep &apos00:1d&apos
00:1d.0 USB controller: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller (rev 05)

Okay so it’s an USB device, even though the physical form factor is a mini-PCIE card. It’s common. Does it match lsusb?

# lsusb | grep Broadband
Bus 002 Device 004: ID 413c:8184 Dell Computer Corp. F3607gw v2 Mobile Broadband Module

Not the Bus/Device specification there, which is good: the device number will increase every time you pop something in/out of the port, so it’s not persistent across reboots at all. What it uses is the path to the device standing by USB ports, which is a tad more complex, but basically means it matches /sys/bus/usb/devices/2-1.6:1.6/ (I don’t pretend to know how the thing works exactly, but it describe to which physical port the device is connected).

In my laptop’s case, the situation is actually quite nice: I cannot move either the WLAN or WWAN device on a different slot so the name assigned by the slot is persistent as well as predictable. But what if you’re on a desktop with an add-on WLAN card? What happens if you decide to change your video card, with a more powerful one that occupies the space of two slots, one of which happen to be the place where you WLAN card is? You move it, reboot and .. you just changed the interface name! If you’ve been using Network Manager, you’ll just have to reconfigure the network I suppose.

Let’s take a different example. My laptop, with its integrated WWAN card, is a rare example; most people I know use USB “keys”, as the providers give them away for free, at least in Italy. I happen to have one as well, so let me try to plug it in one of the ports of my laptop:

# lsusb | grep modem
Bus 002 Device 014: ID 12d1:1436 Huawei Technologies Co., Ltd. E173 3G Modem (modem-mode)
# ifconfig -a | grep ww
wwp0s29u1u2i1: flags=4098<BROADCAST,MULTICAST>  mtu 1500
wwp0s29u1u6i6: flags=4098<BROADCAST,MULTICAST>  mtu 1500

Okay great this is a different USB device, connected to the same USB controller as the onboard one, but at different ports, neat. Now, what if I had all my usual ports busy, and I decided to connect it to the USB3 add-on ExpressCard I got on the laptop?

# lsusb | grep modem
Bus 003 Device 004: ID 12d1:1436 Huawei Technologies Co., Ltd. E173 3G Modem (modem-mode)
# ifconfig -a | grep ww
wwp0s29u1u6i6: flags=4098<BROADCAST,MULTICAST>  mtu 1500
wws1u1i1: flags=4098<BROADCAST,MULTICAST>  mtu 1500

What’s this? Well, the USB3 controller provides slot information, so udev magically uses that to rename the interface, so it avoids using the otherwise longer wwp6s0u1i1 name (the USB3 controller is on the PCI bus 6).

Let’s go back to the on-board ports:

# lsusb | grep modem
Bus 002 Device 016: ID 12d1:1436 Huawei Technologies Co., Ltd. E173 3G Modem (modem-mode)
# ifconfig -a | grep ww
wwp0s29u1u3i1: flags=4098<BROADCAST,MULTICAST>  mtu 1500
wwp0s29u1u6i6: flags=4098<BROADCAST,MULTICAST>  mtu 1500

Seems the same, but it’s not. Now it’s u3 not u2. Why? I used a different port on the laptop. And the interface name changed. Yes, any port change will produce a different interface name, predictably. But what happens if the kernel decides to change the way the ports are enumerated? What happens if the USB 2 driver is buggy and is supposed to provide slot information, and they fix it? You got it, even in these cases, the interface names are changed.

I’m not saying that the kernel naming scheme is perfect. But if you’re expected to always only have an Ethernet port, a WLAN card and a WWAN USB stick, with it you’ll be sure to have eth0, wlan0 and wwan0, as long as the drivers are not completely broken as they are now (like if the WLAN is appearing as eth1), and as long as you don’t muck with the interface names in userspace.

Next up, I’ll talk about the MAC addresses based naming and my personal preference when setting up servers and routers. Have fun in the mean time figuring out what your interface names will be.

10gen released a critical update for mongoDB 2.4.0 which affected queries on secondaries, you should upgrade asap. The python mongo driver followed the 2.4.x releases and got bumped to 2.5 this week-end. I am pleased to announce that I took the chance to add the kerberos authentication support to both ebuilds while bumping them.

pymongo-2.5

• GSSAPI (Kerberos) authentication
• SSL certificate validation with hostname matching
• Delegated and role based authentication

Seems like the only thing everybody has to comment on my previous post was to ask me why I haven’t used $this, $that and ${the kitchen sink}. Or to be precise they asked about cfengine, chef and bcfg2. I have to say I don’t really like being forced into justifying myself but at this point I couldn’t just avoid answering, or I would keep getting the same requests over and over again.

So first of all, why a configuration management system? I have three production vservers at IOS (one is this, another is xine, and another is a customer’s of mine). I have a standby backup server at OVH. And then there’s excelsior, which has four “spacedocks” (containers that I use for building binpkgs for the IOS servers), three tinderbox (but only two usually running), and a couple of “testing” containers (for x32 and musl), beside the actual container I use in it to maintain stuff.

That’s a lot of systems, and while they are very similar between themselves, they are not identical. To begin with, they are in three different countries. And they us three different CPUs. And this is without adding the RaspberryPi I set up with the weather station for a friend of mine. The result is that trying to maintain all those systems manually is a folly, even though I already reduced the number of hosts, since the print shop customer – the one I wrote so often about – moved on and found someone else to pick up their sysadmin tasks (luckily for both of us, since it was a huge time sink).

But the reason why I focused almost exclusively onto Puppet is easy to understand: people I know have been using it for a while. Even though this might sound stupid, I do follow the crowd of friends of mine when I have to figure out what to use. This is because the moment when I have no idea how to do something, it’s easier to ask to a friend than going through the support chain at the upstream project. Gentoo infra people are using and working on Puppet, so that’s a heavy factor to me. I don’t know why they chose puppet but at this point I really don’t care.

But there is another thing, a lesson I learned with Munin: I need to judge the implementation language. The reason is simple, and that’s that I’ll find bugs, for sure. I have this bad knack at finding bugs in stuff I use… which is the main reason why I got interested in open source development: I could then fix the bugs I found! But to do so I have to understand what’s written. And even though learning Perl was easy, understanding Munin’s code… was, and is, tricky. I was able to get some degree of stuff done. Puppet being written in Ruby is a positive note.

I know, chef is also written in Ruby. But I do have a reason to not wanting to deal with chef: its maintainer in Gentoo. Half the bugs I find have to do with the way things are packaged, which is the reason why I became a developer in the first place. This means though that I have to be able to collaborate with the remaining developers, and sometimes that’s just not possible. Sometimes it’s due to upstream developers but in the case of chef the problem is the Gentoo developer who’s definitely not somebody I want to work with, since he’s been “fiddling” with Ruby ebuilds for chef messing up a lot of the work that the Ruby team, me included, kept pouring to improve the quality of the Ruby packages.

So basically these are the reason why I decided to start using Puppet and writing Puppet modules.

So I’ve finally bit the bullet and decided to look into installing and setting up Puppet to manage the configuration of my servers. The reason is to be found in my transfer to Dublin as I expect I won’t have the same time I had before, and that means that any streamlining in the administration of my servers is a net improvement.

In particular, just the other day I spent a lot of time fighting just to set up SSL properly on the servers, and I kept scp’ing files around — it was obvious I wasn’t doing it right.

It goes deeper than this though; since Puppet is obviously trying to get me to standardize the configurations between different servers, I’ve ended up uncovering a number of situations where the configuration of different servers was, well, different. Most of the times without a real reason. For instance, the Munin plugins configured didn’t match, even those that are not specific to a service — of three vservers, one uses PostgreSQL, another uses MySQL and the third, being the standby backup for the two, has both.

Certainly there’s a conflict between your average Gentoo Linux way to do things and the way Puppet expects things to be done. Indeed, the latter requires you to make configurations very similar, while the former tends to make you install each system like its own snowflake — but if you are even partially sane, you would know that to manage more than one Gentoo system, you’ll have to at least standardize some configurations.

The other big problem with using Puppet on Gentoo is that there is a near-showstoppper lack of modules that support our systems. While Theo and Adrien are maintaining a very nice Portage module, there is nothing that allows us to set the OpenRC oldnet-style network configuration, for instance. For other services, often times the support is written with only CentOS or Debian in mind, and the only way to get them to work in Gentoo is to fix the module.

To solve this problem, I started submitting pull requests to modules such as timezone and ntp so that they work on Gentoo. It’s usually relatively easy to do, but it can get tricky, when the CentOS and Gentoo way to set something up are radically different. By the way, the ntp module is sweet because finally I can no longer forget that we have two places to set the NTP server pools.

I also decided to create a module to fit in whatever is Gentoo-specific enough, although this is not yet the kind of stuff you want to rely upon forever — it would have to be done through a real parsed file to set it up properly. On the other hand it allows me set up all my servers’ networks, so it should be okay. And another module allows me to set environment variables on different systems.

You can probably expect me to publish a few more puppet modules – and editing even more – in the next few weeks while I transition as much configuration as I can from custom files to Puppet. In particular, but that’s worth of a separate blog post, I’ll have to work hard to get a nice, easy, and dependable Munin module.

A few months ago, I pointed out what was coming with this release and did an update of this cooking 2.4.0 later. Yesterday, 10gen announced the release of the new stable branch of mongoDB v2.4.0. Instead of talking about it again, I’ll focus on what this release brings to Gentoo users as I’m glad to announce that it’s already available in portage.

SSL support

First of all, I think it was a good time to close bug #421289 and finally enable the SSL support via the ssl USE flag. I’ll support it as much as upstream does, so don’t expect some big magic about it.

Shared client library

Since this has always been a mess, I also added the sharedclient USE flag so that users who really need the client shared library can toggle its installation easily. This also permits me to isolate possible problems from the main ebuild.

Upgrading to 2.4

This is seamless unless you’re running a sharded cluster ! In this case, take great care of what you do and note that the upgrade is only possible if your cluster is running v2.2 ! Please read with care the upgrade plan.

When I’ve wanted to play in some new areas lately, it’s been a real frustration because Gentoo hasn’t had a complete set of packages ready in any of them. I feel like these are some opportunities for Gentoo to be awesome and gain access to new sets of users (or at least avoid chasing away existing users who want better tools):

• Data science. Package Hadoop. Package streaming options like Storm. How about related tools like Flume? RabbitMQ is in Gentoo, though. I’ve heard anecdotally that a well-optimized Hadoop-on-Gentoo installation showed double-digit performance increases over the usual Hadoop distributions (i.e., not Linux distributions, but companies specializing in providing Hadoop solutions). Just heard from Tim Harder (radhermit) than he’s got some packages in progress for a lot of this, which is great news.
• DevOps. This is an area where Gentoo historically did pretty well, in part because our own infrastructure team and the group at the Open Source Lab have run tools like CFEngine and Puppet. But we’re lagging behind the times. We don’t have Jenkins or Travis. Seriously? Although we’ve got Vagrant packaged, for example, we don’t have Veewee. We could be integrating the creation of Vagrant boxes into our release-engineering process.
• Relatedly: Monitoring. Look for some of the increasingly popular open-source tools today, things like Graphite, StatsD, Logstash, Lumberjack, ElasticSearch, Kibana, Sensu, Tasseo, Descartes, Riemann. None of those are there.
• Cloud. Public cloud and on-premise IaaS/PaaS. How about IaaS: OpenStack, CloudStack, Eucalyptus, or OpenNebula? Not there, although some work is happening for OpenStack according to Matthew Thode (prometheanfire). How about a PaaS like Cloud Foundry or OpenShift? Nope. None of the Netflix open-source tools are there. On the public side, things are a bit better — we’ve got lots of AWS tools packaged, even stretching to things like Boto. We could be integrating the creation of AWS images into our release engineering to ensure AWS users always have a recent, official Gentoo image.
• NoSQL. We’ve got a pretty decent set here with some holes. We’ve got Redis, Mongo, and CouchDB not to mention Memcached, but how about graph databases like Neo4j, or other key-value stores like Riak, Cassandra, or Voldemort?
• Android development. Gentoo is perfect as a development environment. We should be pushing it hard for mobile development, especially Android given its Linux base. There’s a couple of halfhearted wiki pages but that does not an effort make. If the SDKs and related packages are there, the docs need to be there too.

Where does Gentoo shine? As a platform for developers, as a platform for flexibility, as a platform to eke every last drop of performance out of a system. All of the above use cases are relevant to at least one of those areas.

I’m writing this post because I would love it if anyone else who wants to help Gentoo be more awesome would chip in with packaging in these specific areas. Let me know!

Update: Michael Stahnke suggested I point to some resources on Gentoo packaging, for anyone interested, so take a look at the Gentoo Development Guide. The Developer Handbook contains some further details on policy as well as info on how to get commit access by becoming a Gentoo developer.

i think i’ve sorted out some of my desktop font issues, and created a few more in the process.

for a long time, i’ve had to deal with occasionally jagged, hard-to-read fonts when viewing webpages, because i ran my xfce desktop without any font antialiasing.

i’ve always hated the way modern desktop environments try to “fool” my eyes with antialiasing and subpixel hinting to convince me that a group of square pixels can be smoothed into round shapes. turning off antialiasing tends to make the rounder fonts, especially serif fonts, look pretty bad at large sizes, as seen here:

my preferred font for the desktop and the web is verdana, which looks pretty good without antialiasing. but most websites use other fonts, so rather than force one size of verdana everywhere (which causes flow/layout issues), i turned on antialiasing for my entire desktop, including my preferred browser, and started disabling antialiasing where needed.

before and after font settings:

i tried the infinality patchset for freetype, but unfortunately none of the eselect configurations produced the crisply rounded antialiased text the patches are known for. i rebuilt freetype without the patchset, and went into /etc/fonts to do some XML hacking.

while eselect-fontconfig offers painless management of existing presets, the only way to customize one’s setup is to get into nitty-gritty text editing, and font configs are in XML format. this is what i ended up with:

$ cat ~/.fonts.conf

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<match target="font">
    <edit name="antialias" mode="assign">
        <bool>false</bool>
    </edit>
</match>
<match target="font" >
    <test name="size" qual="any" compare="more">
        <double>11</double>
    </test>
    <edit name="antialias" mode="assign">
        <bool>true</bool>
    </edit>
</match>
<match target="font" >
    <test name="pixelsize" qual="any" compare="more">
        <double>16</double>
    </test>
    <edit name="antialias" mode="assign">
        <bool>true</bool>
    </edit>
</match>
<match target="pattern">
    <test qual="any" name="family"><string>Helvetica</string></test>
    <edit name="antialias" mode="assign">
      <bool>true</bool>
    </edit>
</match>

let’s step through the rules:

first, all antialiasing is disabled. then, any requested font size over 11, or anything that would display more than 16 pixels high, is antialiased. finally, since the common helvetica font really needs to be antialiased at all sizes, a rule turns that on. in theory, that is — firefox and xfce both seem to be ignoring this. unless antialiasing really is enabled at the smallest sizes with no visible effect, since there are only so many pixel spaces available at that scale to “fake” rounded corners.

a test webpage shows the antialiasing effect on different fonts and sizes:

besides the helvetica issue, there are a few xfce font display problems. xfce is known for mostly ignoring the “modern” xorg font config files, and each app in the desktop environment follows its own aliasing and hinting rules. gvim’s monospace font is occasionally antialiased, resulting in hard-to-read code. the terminal, which uses the exact same font and size, is not antialiased, since it has its own control for text display.

the rest of the gtk+ apps in the above screenshot are size 10 verdana, so they have no antialiasing, being under the “size 11″ rule. firefox doesn’t always obey the system’s font smoothing and hinting settings, even with the proper options in about:config set. unlike user stylesheets, there’s no way to enforce desktop settings with something like !important CSS code. i haven’t found any pattern in what firefox ignores or respects.

also, i haven’t found a workable fontconfig rule that enables antialiasing only for specific fonts at certain sizes. i’m not sure it’s even possible to set such a rule, despite putting together well-formed XML to do just that.

* * *

to sum up: font management on linux can be needlessly complicated, even if you don’t have special vision needs. my environment is overall a bit better, but i’m not ready to move entirely to antialiased text, not until it’s less blurry. i need crispy, sharp text.

fonts on my android phone’s screen look pretty good despite the antialiasing used everywhere, but the thing’s pixel density is so much higher than laptop and desktop LCDs that the display server doesn’t need to resort to complicated smoothing/hinting techniques to achieve that look.

as a general resource, the arch linux wiki page has very useful information on font configuration. there are some great ideas in there, even if they don’t all work on my system. the gentoo linux wiki page on fontconfig is a more basic; i didn’t use anything from it.

Ladies and gentlemen!

If you happen to be involved in using/developing haskell-powered software you might like to answer our poll on that matter.

Thanks in advance!

Today I’ll be masking PostgreSQL 8.3 for removal. If you haven’t already, you should move to a more recent version of PostgreSQL.