Gentoo Logo
Gentoo Logo Side
Gentoo Spaceship

Contributors:
. Aaron W. Swenson
. Agostino Sarubbo
. Alec Warner
. Alex Alexander
. Alex Legler
. Alexey Shvetsov
. Alexis Ballier
. Alexys Jacob
. Amadeusz Żołnowski
. Andreas K. Hüttel
. Andreas Proschofsky
. Anthony Basile
. Arun Raghavan
. Bernard Cafarelli
. Bjarke Istrup Pedersen
. Brent Baude
. Brian Harring
. Christian Faulhammer
. Christian Ruppert
. Christopher Harvey
. Chí-Thanh Christopher Nguyễn
. Daniel Gryniewicz
. David Abbott
. Denis Dupeyron
. Detlev Casanova
. Diego E. Pettenò
. Domen Kožar
. Donnie Berkholz
. Doug Goldstein
. Eray Aslan
. Erik Mackdanz
. Fabio Erculiani
. Gentoo Haskell Herd
. Gentoo Monthly Newsletter
. Gentoo News
. Gilles Dartiguelongue
. Greg KH
. Hanno Böck
. Hans de Graaff
. Ian Whyman
. Ioannis Aslanidis
. Jan Kundrát
. Jason Donenfeld
. Jauhien Piatlicki
. Jeffrey Gardner
. Jeremy Olexa
. Joachim Bartosik
. Johannes Huber
. Jonathan Callen
. Jorge Manuel B. S. Vicetto
. Joseph Jezak
. Kenneth Prugh
. Kristian Fiskerstrand
. Lance Albertson
. Liam McLoughlin
. LinuxCrazy Podcasts
. Luca Barbato
. Luis Francisco Araujo
. Mark Loeser
. Markos Chandras
. Mart Raudsepp
. Matt Turner
. Matthew Marlowe
. Matthew Thode
. Matti Bickel
. Michael Palimaka
. Michal Hrusecky
. Michał Górny
. Mike Doty
. Mike Gilbert
. Mike Pagano
. Nathan Zachary
. Ned Ludd
. Nirbheek Chauhan
. Pacho Ramos
. Patrick Kursawe
. Patrick Lauer
. Patrick McLean
. Pavlos Ratis
. Paweł Hajdan, Jr.
. Peter Wilmott
. Petteri Räty
. Piotr Jaroszyński
. Rafael Goncalves Martins
. Raúl Porcel
. Remi Cardona
. Richard Freeman
. Robin Johnson
. Ryan Hill
. Sean Amoss
. Sebastian Pipping
. Steev Klimaszewski
. Stratos Psomadakis
. Sune Kloppenborg Jeppesen
. Sven Vermeulen
. Sven Wegener
. Theo Chatzimichos
. Thomas Kahle
. Tiziano Müller
. Tobias Heinlein
. Tobias Klausmann
. Tom Wijsman
. Tomáš Chvátal
. Victor Ostorga
. Vikraman Choudhury
. Vlastimil Babka
. Zack Medico

Last updated:
February 12, 2016, 16:04 UTC

Disclaimer:
Views expressed in the content published here do not necessarily represent the views of Gentoo Linux or the Gentoo Foundation.


Bugs? Comments? Suggestions? Contact us!

Powered by:
Planet Venus

Welcome to Planet Gentoo, an aggregation of Gentoo-related weblog articles written by Gentoo developers. For a broader range of topics, you might be interested in Gentoo Universe.

February 10, 2016
Denis Dupeyron a.k.a. calchan (homepage, bugs)
It is GSoC season again (February 10, 2016, 17:54 UTC)

Google Summer of Code 2016 is starting.

If you are a student please be patient. Your time will come soon, at which point we’ll be able to answer all your questions. In this initial phase the audience is project mentors. Note that you do not need to be a Gentoo developer to be a mentor.

While we are finalizing the application, we need all of you to submit your project ideas before the end of next week (February 19th). To do so you should go to this year’s idea page and follow the instructions in the “Ideas” section. If you proposed an idea last year and would like to propose it again for this year, you can look it up and add it. Or you can just tell us and we will do it for you. Don’t hesitate to add an idea even it isn’t totally fleshed out; we will help you fill in the blanks. A project represents 3 months of full-time work for a student. A good rule of thumb is that it should take you between 2 and 4 weeks to complete depending on how expert you are in the field. You can also have a look at last year’s idea page for examples.

If you would like to be a mentor for this year please tell us sooner rather than later. You can be a mentor following up with students while they’re working on their project, or you can be an expert-mentor in a specific field who will be called in to advise on an as-needed basis (or maybe never). We will also need people to help us review project proposals. In case you can help in any capacity don’t hesitate to contact us. GSoC is seriously fun!

If you want to reach us or have any questions about any of the above, the easiest way is to ping Calchan or rafaelmartins in the #gentoo-soc channel on Freenode.

February 08, 2016
Michał Górny a.k.a. mgorny (homepage, bugs)
A quick note on portable shebangs (February 08, 2016, 12:57 UTC)

While at first shebangs may seem pretty obvious and well supported, there is a number of not-so-well-known portability issues affecting them. Only during my recent development work, I have hit more than one of them. For this reason, I’d like to write a quick note summarizing how to stay on the safe side and keep your scripts working across various systems.

Please note I will only cover the basic solution to the most important portability issues. If you’d like to know more about shebang handling in various systems, I’d like to recommend you an excellent article ‘The #! magic, details about the shebang/hash-bang mechanism on various Unix flavours’ by Sven Mascheck.

So, in order to stay portable you should note that:

  1. Many systems (Linux included!) have limits on shebang length. If you exceed this length, the kernel will cut the shebang in the middle of a path component, and usually try to execute the script with the partial path! To stay safe you need to keep shebang short. Since you can’t really control where the programs are installed (think of Prefix!), you should always rely on PATH lookups.
  2. Shebangs do not have built-in PATH lookups. Instead, you have to use the /usr/bin/env tool which performs the lookup on its argument (the exact path is mostly portable, with a few historical exceptions).
  3. Different systems split parameters in shebangs differently. In particular, Linux splits on the first space only, passing everything following it as a single parameter. To stay portable, you can not pass more than one parameter, and it can not contain whitespace. Which — considering the previous points made — means the parameter is reserved for program name passed to env, and you can not pass any actual parameters.
  4. Shebang nesting (i.e. referencing an interpreted script inside a shebang) is supported only by some systems, and only to some extent. For this reason, shebangs need to reference actual executable programs. However, using env effectively works around the issue since env is the immediate interpreter.

A few quick examples:

#!/usr/bin/env python  # GOOD!

#!python  # BAD: won't work

#!/usr/bin/env python -b  # BAD: it may try to spawn program named 'python -b'

#!/usr/bin/python  # BAD: absolute path is non-portable, also see below

#!/foo/bar/baz/usr/bin/python  # BAD: prefix can easily exceed length limit

#!/usr/lib/foo/foo.sh  # BAD: calling interpreted scripts is non-portable

January 31, 2016
Gentoo at FOSDEM 2016 (January 31, 2016, 13:00 UTC)

Gentoo Linux was present at this year's Free and Open Source Developer European Meeting (FOSDEM). For those not familiar with FOSDEM it is a conference that consists of more than 5,000 developers and more than 600 presentations over a two-day span at the premises of the Université libre de Bruxelles. The presentations are both streamed directly and recorded making it available to browse the archive once published.

Hanno Böck, a name mostly heard in relation to the fuzzing project, was the only Gentoo Developer presenting a talk this year on the very important subject of security and how Gentoo can be used as a framework for running Address Sanitizer to detect security bugs: "Can we run C code and be safe?: A Linux system protected with Address Sanitizer".

For the first time in many years Gentoo had a stand this year where we handed out buttons and stickers in addition to a LiveDVD.

Gentoo Boot

The Gentoo Ten team created a hybrid amd64/x86 "FOSDEM 2016 Special Edition" release for our user's benefit (thanks likewhoa!), and 200 DVDs were printed of which 155 were already distributed to potential users by the end of day one. A posters on the stand succinctly listed all the packages included on the LiveDVD with some highlights of packages more familiar to some users, something that also highlights one of the benefits of rolling release distributions in that the versions are up to date with upstream releases.

Gentoo DVD package list

If the LiveDVD is used on USB instead of the handed out DVDs it also offers the option of using persistence to store changes on the USB. It uses aufs to layer a read-write file system on top of a read-only squashfs compressed file system. This is great, because it allows you to make changes to the livedvd and have those changes appear on future reboots.

As mentioned in a blog post by dilfridge the stand also attracted attention due to a comment involving Gentoo Linux by Lennart Poettering in his keynote speech as a distribution that doesn't use systemd by default. This fit nicely with one of our banners at the stand; "Gentoo Linux | Works even without systemd | choice included".

gentoo-choice

There was a lot of positive feedback from various users and the stand functioned very nicely as a meeting place for the various people and the atmosphere was good throughout the conference.

10 fosdem-booth

As has become a tradition there was also a Gentoo dinner again this year amongst developers and users (thanks xaviermiller), a nice way to meet up and discuss everything in a relaxing setting.

January 30, 2016
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)
Gentoo at FOSDEM: Posters (systemd, arches) (January 30, 2016, 15:24 UTC)

Especially after Lennart Poettering made some publicity for Gentoo Linux in his keynote talk (unfortunately I missed it due to other commitments :), we've had a lot of visitors at our FOSDEM booth. So, because of popular demand, here are again the files for our posters. They are based on the great "Gentoo Abducted" design by Matteo Pescarin.  Released under CC BY-SA 2.5 as the original. Enjoy!



PDF SVG


PDF SVG

January 29, 2016
Matthew Thode a.k.a. prometheanfire (homepage, bugs)
Stage4 tarballs, minimal and cloud (January 29, 2016, 05:03 UTC)

Where are they

The tarballs can be found in the normal place.

Minimal

This is meant to be just what you need to boot, the disk won't expand itself, it won't even get networking info or set any passwords for you (no default password).

This tarball is suposed to be the base you generate more complex images from, it is what is going to be used by Openstack's diskimage-builder.

The primary things it does is get you a kernel, bootloader and sshd.

stage4-minimal spec

Cloud

This was primarilly targeted for use with openstack but it should work with amazon as well, both use cloud-init.

Network interfaces are expected to use dhcp, a couple of other useful things are installed as well, syslog, logrotate, etc.

By default cloud-init will take data (keys mainly) and set them up for the 'gentoo' user.

stage4-cloud spec

Next

I'll be posting about the work being done to take these stages and build bootable images. At the momebt I do have images available here.

openstack images

January 26, 2016
Hanno Böck a.k.a. hanno (homepage, bugs)

GentooAddress Sanitizer is a remarkable feature that is part of the gcc and clang compilers. It can be used to find many typical C bugs - invalid memory reads and writes, use after free errors etc. - while running applications. It has found countless bugs in many software packages. I'm often surprised that many people in the free software community seem to be unaware of this powerful tool.

Address Sanitizer is mainly intended to be a debugging tool. It is usually used to test single applications, often in combination with fuzzing. But as Address Sanitizer can prevent many typical C security bugs - why not use it in production? It doesn't come for free. Address Sanitizer takes significantly more memory and slows down applications by 50 - 100 %. But for some security sensitive applications this may be a reasonable trade-off. The Tor project is already experimenting with this with its Hardened Tor Browser.

One project I've been working on in the past months is to allow a Gentoo system to be compiled with Address Sanitizer. Today I'm publishing this and want to allow others to test it. I have created a page in the Gentoo Wiki that should become the central documentation hub for this project. I published an overlay with several fixes and quirks on Github.

I see this work as part of my Fuzzing Project. (I'm posting it here because the Gentoo category of my personal blog gets indexed by Planet Gentoo.)

I am not sure if using Gentoo with Address Sanitizer is reasonable for a production system. One thing that makes me uneasy in suggesting this for high security requirements is that it's currently incompatible with Grsecurity. But just creating this project already caused me to find a whole number of bugs in several applications. Some notable examples include Coreutils/shred, Bash ([2], [3]), man-db, Pidgin-OTR, Courier, Syslog-NG, Screen, Claws-Mail ([2], [3]), ProFTPD ([2], [3]) ICU, TCL ([2]), Dovecot. I think it was worth the effort.

I will present this work in a talk at FOSDEM in Brussels this Saturday, 14:00, in the Security Devroom.

January 25, 2016
Michał Górny a.k.a. mgorny (homepage, bugs)
Mangling shell options in ebuilds (January 25, 2016, 10:46 UTC)

A long time ago eutils.eclass was gifted with a set of terribly ugly functions to push/pop various variables and shell options. Those functions were written very badly, and committed without any review. As a result, a number of eclasses and ebuilds are now using that code without even understanding how bad it is.

In this post, I would like to shortly summarize how to properly and reliably save states of shell options. While the resulting code is a little bit longer than use of e*_push and e*_pop functions, it is much more readable, does not abuse eval, does not abuse global variables and is more reliable.

Preferable solution: subshell scope

Of course, the preferable way of altering shell options is to do that in a subshell. This is the only way that reliably isolates the alterations from parent ebuild environment. However, subshells are rarely desired — so this is something you’d rather reuse if it’s already there, rather than introducing just for the sake of shell option mangling.

Mangling shopt options

Most of the ‘new’ bash options are mangled using shopt builtin. In this case, the -s and -u switches are used to change the option state, while the -p option can be used to get the current value. The current value is output in the form of shopt command syntax that can be called directly to restore the previous value.

my_function() {
	local prev_shopt=$(shopt -p nullglob)
	# prev_shopt='shopt -u nullglob' now
	shopt -s nullglob
	# ...
	${prev_shopt}
}

Mangling set options

The options set using the set builtin can be manipulated in a similar way. While the builtin support both short and long options, I strongly recommend using long options for readability. In fact, the long option names can be used through shopt with the additional -o parameter.

my_function() {
	local prev_shopt=$(shopt -p -o noglob)
	# prev_shopt='set +o noglob' now
	set -o noglob  # or shopt -s -o noglob
	# ...
	${prev_shopt}
}

Mangling umask

The umask builtin returns the current octal umask when called with no parameters. Furthermore, the -p parameter can be used to get full command for use alike shopt -p output.

my_function() {
	local prev_umask=$(umask)
	# prev_umask=0022 now
	umask 077
	# ...
	umask "${prev_umask}"
}

alternative_function() {
	local prev_umask=$(umask -p)
	# prev_umask='umask 0022' now
	umask 077
	# ...
	${prev_umask}
}

Mangling environment variables

The eutils hackery went as far as to reinvent local variables using… global stacks. Not that it makes any sense. Whenever you want to change variable’s value, attributes or just unset it temporarily, just use local variables. If the change needs to apply to part of a function, create a sub-function and put the local variable inside it.

While at it, please remember that bash does not support local functions. Therefore, you need to namespace your functions to avoid collisions and unset them after use.

my_function() {
	# unset FOO in local scope (this also prevents it from being exported)
	local FOO
	# 'localize' bar for modifications, preserving value
	local bar="${bar}"

	#...

	my_sub_func() {
		# export LC_ALL=POSIX in function scope
		local -x LC_ALL=POSIX
		#...
	}
	my_sub_func
	# unset the function after use
	unset -f my_sub_func
}

Update: mangling shell options without a single subshell

(added on 2016-01-28)

izabera has brought it to my attention that the shopt builtin supports -q option to suppress output and uses exit statuses to return the original flag state. This makes it possible to set and unset the flags without using a single subshell or executing returned commands.

Since I do not expect most shell script writers to use such a long replacement, I present it merely as a curiosity.

my_setting_function() {
	shopt -q nullglob
	local prev_shopt=${?}
	shopt -s nullglob

	#...

	[[ ${prev_shopt} -eq 0 ]] || shopt -u nullglob
}

my_unsetting_function() {
	shopt -q extquote
	local prev_shopt=${?}
	shopt -u extquote

	#...

	[[ ${prev_shopt} -eq 0 ]] && shopt -s extquote
}

January 24, 2016
Jan Kundrát a.k.a. jkt (homepage, bugs)
Trojita 0.6 is released (January 24, 2016, 11:14 UTC)

Hi all,
we are pleased to announce version 0.6 of Trojitá, a fast Qt IMAP e-mail client. This release brings several new features as well as the usual share of bugfixes:

  • Plugin-based infrastructure for the address book, which will allow better integration with other applications
  • Usability improvements in the message composer on several fronts
  • Better keyboard-only usability for those of us who do not touch mouse that often
  • More intuitive message tagging, and support for standardized actions for junk mail
  • Optional sharing of authentication data between IMAP and SMTP
  • Change to using Qt5 by default. This is the last release which still supports Qt4.
  • Improved robustness on unstable network connections
  • The old status bar is now gone to save screen real estate
  • IMAP interoperability fixes
  • Speed improvements

This release has been tagged in git as "v0.6". You can also download a tarball (GPG signature). Prebuilt binaries for multiple distributions are available via the OBS, and so is a Windows installer.

This release is named after the Aegean island Λέσβος (Lesvos). Jan was there for the past five weeks, and he insisted on mentioning this challenging experience.

The Trojitá developers

January 22, 2016
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)
Please test www-apache/mod_perl-2.0.10_pre201601 (January 22, 2016, 18:39 UTC)

We're trying to get both Perl 5.22 and Apache 2.4 stable on Gentoo these days. One thing that would be really useful is to have a www-apache/mod_perl that works with all current Perl and Apache versions... and there's a candidate for that: a snapshot of what is hopefully going to be mod_perl-2.0.10 pretty soon. So...

Please keyword (if necessary) and test www-apache/mod_perl-2.0.10_pre201601!!! Feedback for all Perl and Apache versions is very much appreciated. Gentoo developers can directly edit our compatibility table with the results, everyone else please comment on this blog post or file bugs in case of problems!

Please always include exact www-servers/apache, dev-lang/perl, and www-apache/mod_perl versions!

January 18, 2016
Matthew Thode a.k.a. prometheanfire (homepage, bugs)
Of OpenStack and SSL (January 18, 2016, 06:00 UTC)

SSL in vanila OpenStack

The nature of OpenStack projects is largely like projects in Gentoo. Even though they are all under the OpenStack umbrella that doesn't mean they all have to work the same, or even work together.

For instance, nova has the ability to do ssl itself, you can define a CA and public/private keypair. Glance (last time I checked) doesn't do ssl yourself so you must offload it. Other service might do ssl themselves but not in the same way nova does it.

This means that the most 'standard' setup would be to not run ssl, but this isn't exactly desirable. So run a ssl reverse proxy.

Basic Setup

  • OpenStack services are set up on one host (just in this example).
  • OpenStack services are configed to listen on localhost only.
  • Public, Internal and Admin URLs need to be defined with https.
  • Some tuning needs to be done so services work properly, primarily to glance and nova-novnc.
  • Nginx is used as the reverse proxy.

Configs and Tuning

General Config for All Services/Sites

This is the basic setup for each of the openstack services, the only difference between them will be what goes in the location subsection.

server {
    listen LOCAL_PUBLIC_IPV4:PORT;
    listen [LOCAL_PUBLIC_IPV6]:PORT;
    server_name = name.subdomain.example.com;
    access_log /var/log/nginx/keystone/access.log;
    error_log /var/log/nginx/keystone/error.log;

    ssl on;
    ssl_certificate /etc/nginx/ssl/COMBINED_PUB_PRIV_KEY.pem;
    ssl_certificate_key /etc/nginx/ssl/COMBINED_PUB_PRIV_KEY.pem;
    add_header Public-Key-Pins 'pin-sha256="PUB_KEY_PIN_SHA"; max-age=2592000; includeSubDomains';
    ssl_dhparam /etc/nginx/params.4096;
    resolver TRUSTED_DNS_SERVER;
    resolver_timeout 5s;
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/nginx/ssl/COMBINED_PUB_PRIV_KEY.pem;
    add_header X-XSS-Protection "1; mode=block";
    add_header Content-Security-Policy "default-src 'self' https: wss:;";
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    location / {
        # this changes depending on the service
        proxy_pass http://127.0.0.1:PORT;
    }
}

Keystone and Uwsgi

It turns out keystone has switched to uwsgi for it's service backend. This is good because it means we can have the web server connect to that, no more trying to do it all by itself. I'll leave the setting up of uwsgi itself as an exercise to the reader :P

This config has a few extra things, but it is currently what I know to be 'secure' (similiar config on this blog gets an A+ on all those ssl test things). It's the last location piece that changes the most between services.

location / {
    uwsgi_pass unix:///run/uwsgi/keystone_admin.socket;
    include /etc/nginx/uwsgi_params;
    uwsgi_param SCRIPT_NAME admin;
}

Glance

Glance just needs one thing on top of the general proxying that it needs. It needs client_max_body_size 0; in the main server stanza so that you can upload images without being cut off at some low size.

client_max_body_size 0;
location / {
    proxy_pass http://127.0.0.1:9191;
}

Nova

The serviecs for nova just need the basic proxy_pass line. The only exception is novnc, it needs some proxy headers passed.

location / {
    proxy_pass http://127.0.0.1:6080;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
}

Rabbitmq

Rabbit is fairly simple, you just need to enable ssl and disable the plaintext port (setting up your config of course).

[
    {ssl, [{versions, ['tlsv1.2', 'tlsv1.1']}]},
    {rabbit, [
        {tcp_listeners, []},
        {ssl_listeners, [5671]},
        {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/CA_CERT.pem"},
                       {certfile,  "/etc/rabbitmq/ssl/PUB_KEY.pem"},
                       {keyfile,   "PRIV_KEYKEY.key"},
                       {versions, ['tlsv1.2', 'tlsv1.1']}
                      ]
        }]
    }
].

Openstack Configs

The openstack configs can differ slightly but they are all mostly the same now they are using the same libraries (oslo stuff).

General Config

[keystone_authtoken]
auth_uri = https://name.subdomain.example.com:5000
auth_url = https://name.subdomain.example.com:35357

[oslo_messaging_rabbit]
rabbit_host = name.subdomain.example.com
rabbit_port = 5671
rabbit_use_ssl = True

Nova

osapi_compute_listen = 127.0.0.1
metadata_listen = 127.0.0.1
novncproxy_host = 127.0.0.1
enabled_apis = osapi_compute, metadata
[vnc]
novncproxy_base_url = https://name.subdomain.example.com:6080/vnc_auto.html
# the following only on the 'master' host
vncserver_proxyclient_address = 1.2.3.4
vncserver_listen = 1.2.3.4

[glance]
host = name.subdomain.example.com
protocol = https
api_servers = https://name.subdomain.example.com:9292

[neutron]
url = https://name.subdomain.example.com:9696
auth_url = https://name.subdomain.example.com:35357

Cinder

# api-servers get this
osapi_volume_listen = 127.0.0.1

# volume-servers and api-servers get this
glance_api_servers=https://name.subdomain.example.com:9292

Glance

glance_api_servers=https://name.subdomain.example.com:9292

Glance

# api
bind_host = 127.0.0.1
registry_host = name.subdomain.example.com
registry_port = 9191
registry_client_protocol = https

.

# cache
registry_host = name.subdomain.example.com
registry_port = 9191

.

# registry
bind_host = 127.0.0.1
rabbit_host = name.subdomain.example.com
rabbit_port = 5671
rabbit_use_ssl = True

.

# scrubber
registry_host = name.subdomain.example.com
registry_port = 9191

Neutron

# neutron.conf
bind_host = 127.0.0.1
nova_url = https://name.subdomain.example.com:8774/v2

[nova]
auth_url = https://name.subdomain.example.com:35357

.

# metadata_agent.ini
nova_metadata_ip = name.subdomain.example.com
nova_metadata_protocol = https

January 16, 2016
Diego E. Pettenò a.k.a. flameeyes (homepage, bugs)

You might have seen the word TEXTREL thrown around security or hardening circles, or used in Gentoo Linux installation warnings, but one thing that is clear out there is that the documentation around this term is not very useful to understand why they are a problem. so I've been asked to write something about it.

Let's start with taking apart the terminology. TEXTREL is jargon for "text relocation", which is once again more jargon, as "text" in this case means "code portion of an executable file." Indeed, in ELF files, the .text section is the one that contains all the actual machine code.

As for "relocation", the term is related to dynamic loaders. It is the process of modifying the data loaded from the loaded file to suit its placement within memory. This might also require some explanation.

When you build code into executables, any named reference is translated into an address instead. This includes, among others, variables, functions, constants and labels — and also some unnamed references such as branch destinations on statements such as if and for.

These references fall into two main typologies: relative and absolute references. This is the easiest part to explain: a relative reference takes some address as "base" and then adds or subtracts from it. Indeed, many architectures have a "base register" which is used for relative references. In case of executable code, particularly with the reference to labels and branch destinations, relative references translate into relative jumps, which are relative to the current instruction pointer. An absolute reference is instead a fully qualified pointer to memory, well at least to the address space of the running process.

While absolute addresses are kinda obvious as a concept, they are not very practical for a compiler to emit in many cases. For instance, when building shared objects, there is no way for the compiler to know which addresses to use, particularly because a single process can load multiple objects, and they need to all be loaded at different addresses. So instead of writing to the file the actual final (unknown) address, what gets written by the compiler first – and by the link editor afterwards – is a placeholder. It might sound ironic, but an absolute reference is then emitted as a relative reference based upon the loading address of the object itself.

When the loader takes an object and loads it to memory, it'll be mapped at a given "start" address. After that, the absolute references are inspected, and the relative placeholder resolved to the final absolute address. This is the process of relocation. Different types of relocation (or displacements) exists, but they are not the topic of this post.

Relocations as described up until now can apply to both data and code, but we single out code relocations as TEXTRELs. The reason for this is to be found in mitigation (or hardening) techniques. In particular, what is called W^X, NX or PaX. The basic idea of this technique is to disallow modification to executable areas of memory, by forcing the mapped pages to either be writable or executable, but not both (W^X reads "writable xor executable".) This has a number of drawbacks, which are most clearly visible with JIT (Just-in-Time) compilation processes, including most JavaScript engines.

But beside JIT problem, there is the problem with relocations happening in code section of an executable, too. Since the relocations need to be written to, it is not feasible (or at least not easy) to provide an exclusive writeable or executable access to those. Well, there are theoretical ways to produce that result, but it complicates memory management significantly, so the short version is that generally speaking, TEXTRELs and W^X techniques don't go well together.

This is further complicated by another mitigation strategy: ASLR, Address Space Layout Randomization. In particular, ASLR fully defeats prelinking as a strategy for dealing with TEXTRELs — theoretically on a system that allows TEXTREL but has the address space to map every single shared object at a fixed address, it would not be necessary to relocate at runtime. For stronger ASLR you also want to make sure that the executables themselves are mapped at different addresses, so you use PIE, Position Independent Executable, to make sure they don't depend on a single stable loading address.

Usage of PIE was for a long while limited to a few select hardened distributions, such as Gentoo Hardened, but it's getting more common, as ASLR is a fairly effective mitigation strategy even for binary distributions where otherwise function offsets would be known to an attacker.

At the same time, SELinux also implements protection against text relocation, so you no longer need to have a patched hardened kernel to provide this protection.

Similarly, Android 6 is now disallowing the generation of shared objects with text relocations, although I have no idea if binaries built to target this new SDK version gain any more protection at runtime, since it's not really my area of expertise.

Michał Górny a.k.a. mgorny (homepage, bugs)

The way packages are maintained in Gentoo have been evolving for quite some time already. So far all of that has been happening on top of old file formats which slowly started to diverge from the needs of Gentoo developers, and become partially broken. The concept of herds has become blurry, with confusion in definition between different developers and partial assumption about their deprecation. Maintenance of herd by project has been broken by moving projects to the Wiki. Some projects have stopped using herds, others have been declaring them in metadata.xml in different ways.

The problem has finally reached the Gentoo Council and has been discussed on 2015-10-25 meeting (note: no summary still…). The Council attempted to address different problems by votes, and create a new solution by combining the results of votes. However, finally it decided that it is not possible to create a good specification this way. Instead, the meeting has brought two major points. Firstly, herds are definitely deprecated. Secondly, someone needs to provide a complete, consistent replacement in GLEP form.

This is how GLEP 67 came to be. It was based on results of previous discussion, Council votes and thorough analysis of different problems. It provides a complete, consistent system for maintaining packages and expressing the maintenance information. It has been approved by the Council on 2016-01-10, with two week deadline on preparing to the switch.

Therefore, on 2016-01-24 Gentoo is going to switch to the new maintenance structure described in GLEP 67 completely. The announcement with transition details has been sent already. Instead, I’d like to focus on describing how things are going to work starting from the day GLEP 67 becomes implemented.

Who is going to maintain packages?

Before getting into technical details, GLEP 67 starts by limiting possible package maintainer entries. Until now, metadata files were allowed to list practically any e-mail addresses for package maintainers. From now on, only real people (either developers or proxied maintainers) and projects (meeting the requirements of GLEP 39, in particular having a Wiki page) are allowed to be maintainers. All maintainers are identified by e-mail addresses which are required to be unique (i.e. sharing the same address between multiple projects is forbidden) and registered on bugs.g.o.

This supports the two major goals behind maintainer specifications: bug assignment and responsibility assignment. The former is rather obvious — Bugzilla is the most reliable communication platform for both Gentoo developers and users. Therefore, it is important that the bugs can be assigned to appropriate entities without any issues. The latter aims to address the problem of unclear ‘ownership’ of some packages, and packages maintained by ‘dead’ e-mail aliases.

In other words, from now on we require that for every maintained package in Gentoo, we need to be able to obtain a complete, clear list of people maintaining it (directly and via projects). We no longer accept ‘dumb’ e-mail aliases that make it impossible to distinguish real maintainers from people who are simply following bugs. This gives three important advantages:

  1. we can actually ping the correct people on IRC without having to go through hoops,
  2. we can determine whether a package is actually maintained by someone, rather than assigned to an alias from which nobody reads bug mail anymore,
  3. we can clearly determine who is responsible for a package and who is the appropriate person to acknowledge changes.

Changes for maintainer-needed packages

The new requirements brought a new issue: maintainer-needed@g.o. The specific use of this e-mail alias did not really seem to suit a project. Creating a ‘maintainer-needed project’ would either imply creating a dead entity or assigning actual maintainers to supposedly-unmaintained packages. On the other hand, I did not really want to introduce special cases in the specification.

Instead, I have decided that the best way forward is to remove it. In other words, unmaintained packages now explicitly list no maintainers. The assignment to maintainer-needed@g.o will be done implicitly, and is a rule specific to the Gentoo repository. Other repositories may use different policies for packages with no explicit maintainers (like assigning to the repository owner).

The metadata.xml and projects.xml files

The changes to metadata.xml are really minimal, and backwards compatible. The <herd/> element is no longer used, and will be prohibited. Instead, <maintainer/> elements are going to be used for all kinds of maintainers. There are no incompatible changes in those elements, therefore existing tools will not be broken.

The <maintainer/> element gets a new obligatory type attribute that needs to either be person or project. The latter value may cause tools to look the project up (by e-mail) in projects.xml.

The projects.xml file (unlike past herds.xml) is clearly defined in the repository scope. In particular, the tools must not assume it always comes out of Gentoo repository. Other repositories are allowed to define their own projects (though overriding projects is not allowed), and project lookup needs to respect masters= setting in repositories.

For the Gentoo repository, projects.xml is generated automatically from the Wiki project pages, and distributed both via api.gentoo.org and via the usual repository distribution means (the rsync and git mirrors).

Summary

A quick summary for Gentoo developers of how things look like with GLEP 67:

  1. Only people and projects can maintain packages. If you want to maintain packages in an organized group, you have to create a project — with Wiki page, unique e-mail address and bugs.g.o account.
  2. Only the explicitly listed project members (and subproject members whenever member inheritance is enabled) are considered maintainers of the package. Other people subscribing the e-mail alias are not counted.
  3. Packages with no maintainers have no <maintainer/> elements. The bugs are still implicitly assigned to maintainer-needed@g.o but this e-mail alias is no longer used in metadata.xml.
  4. <herd/> elements are no longer used, <maintainer/>s are used instead.
  5. <maintainer/> requires a new type attribute that takes either person or project value.

January 05, 2016
Sebastian Pipping a.k.a. sping (homepage, bugs)

Demo start:
https://www.youtube.com/watch?v=2A7V3GLWF6U&feature=youtu.be&t=37s

Where “OpenRC 0.19.1 is starting up Gentoo Linux (x86_64)” scrolls into display:
https://www.youtube.com/watch?v=2A7V3GLWF6U&feature=youtu.be&t=1m21s

January 03, 2016

The new year kicks off with two large events with Gentoo participation: The Southern California Linux Expo SCALE14x and FOSDEM 2016, both featuring a Gentoo booth.

SCALE14x

SCALE14x logo

First we have the Southern California Linux Expo SCALE in its 14th edition. The Pasadena Convention Center will host the event this year from January 21 to January 24.

Gentoo will be present as an exhibitor, like in the previous years.

Thanks to the organizers, we can share a special promotional code for attending SCALE with our community, valid for full access passes. Using the code GNTOO on the registration page will get you a 50% discount.

FOSDEM 2016

FOSDEM 2016 logo

Then, on the last weekend of January, we’ll be on the other side of the pond in Brussels, Belgium where FOSDEM 2016 will take place on January 30 and 31.

Located at the Université libre de Bruxelles, it doesn’t just offer interesting talks, but also the finest Belgian beers when the sun sets. :)

This year, Gentoo will also be manning a stand with gadgets, swag, and LiveDVDs.

Booth locations

We’ll update this news item with more detailed information on how to find our booths at both conferences once we have information from the organizers.

January 02, 2016
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
Gentoo Linux on DELL XPS 13 9350 (January 02, 2016, 10:43 UTC)

As I found little help about this online I figured I’d write a summary piece about my recent experience in installing Gentoo Linux on a DELL XPS 13 9350.

UEFI or MBR ?

This machine ships with a NVME SSD so don’t think twice : UEFI is the only sane way to go.

BIOS configuration

I advise to use the pre-installed Windows 10 to update the XPS to the latest BIOS (1.1.7 at the time of writing). Then you need to change some stuff to boot and get the NVME SSD disk discovered by the live CD.

  • Turn off Secure Boot
  • Set SATA Operation to AHCI (will break your Windows boot but who cares)

Live CD

Go for the latest SystemRescueCD (it’s Gentoo based, you won’t be lost) as it’s quite more up to date and supports booting on UEFI. Make it a Live USB for example using unetbootin and the ISO on a vfat formatted USB stick.

NVME SSD disk partitioning

We’ll be using GPT with UEFI. I found that using gdisk was the easiest. The disk itself is found on /dev/nvme0n1. Here it is the partition table I used :

  • 500Mo UEFI boot partition (type EF00)
  • 16Go Swap partition
  • 60Go Linux root partition
  • 400Go home partition

The corresponding gdisk commands :

# gdisk /dev/nvme0n1

Command: o ↵
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y ↵

Command: n ↵
Partition Number: 1 ↵
First sector: ↵
Last sector: +500M ↵
Hex Code: EF00 ↵

Command: n ↵
Partition Number: 2 ↵
First sector: ↵
Last sector: +16G ↵
Hex Code: 8200 ↵

Command: n ↵
Partition Number: 3 ↵
First sector: ↵
Last sector: +60G ↵
Hex Code: ↵

Command: n ↵
Partition Number: 4 ↵
First sector: ↵
Last sector: ↵ (for rest of disk)
Hex Code: ↵

Command: w ↵
Do you want to proceed? (Y/N): Y ↵

No WiFi on Live CD ? no panic

If your live CD is old (pre 4.4 kernel), the integrated broadcom 4350 wifi card won’t be available !

My trick was to use my Android phone connected to my local WiFi as a USB modem which was detected directly by the live CD.

  • get your Android phone connected on your local WiFi (unless you want to use your cellular data)
  • plug in your phone using USB to your XPS
  • on your phone, go to Settings / More / Tethering & portable hotspot
  • enable USB tethering

Running ip addr will show the network card enp0s20f0u2 (for me at least), then if no IP address is set on the card, just ask for one :

# dhcpcd enp0s20f0u2

Et voilà, you have now access to the internet.

Proceed with installation

The only thing to worry about is to format the UEFI boot partition as FAT32.

# mkfs.vfat -F 32 /dev/nvme0n1p1

Then follow the Gentoo handbook as usual for the next steps of the installation process until you arrive to the kernel and the bootloader / grub part.

From this moment I can already say that NO we won’t be using GRUB at all so don’t bother installing it. Why ? Because at the time of writing, the efi-64 support of GRUB was totally not working at all as it failed to discover the NVME SSD disk on boot.

Kernel sources and consideration

The trick here is that we’ll setup the boot ourselves directly from the BIOS later so we only need to build a standalone kernel (meaning able to boot without an initramfs).

EDIT: as of Jan. 10 2016, kernel 4.4 is available on portage so you don’t need the patching below any more !

Make sure you install and use at least a 4.3.x kernel (4.3.3 at the time of writing). Add sys-kernel/gentoo-sources to your /etc/portage/package.keywords file if needed. If you have a 4.4 kernel available, you can skip patching it below.

Patching 4.3.x kernels for Broadcom 4350 WiFi support

To get the broadcom 4350 WiFi card working on 4.3.x, we need to patch the kernel sources. This is very easy to do thanks to Gentoo’s user patches support. Do this before installing gentoo-sources (or reinstall it afterwards).

This example is for gentoo-sources-4.3.3, adjust your version accordingly :

(chroot) # mkdir -p /etc/portage/patches/sys-kernel/gentoo-sources-4.3.3
(chroot) # cd /etc/portage/patches/sys-kernel/gentoo-sources-4.3.3
(chroot) # wget http://ultrabug.fr/gentoo/xps9350/0001-bcm4350.patch

When emerging the gentoo-sources package, you should see the patch being applied. Check that it worked by issuing :

(chroot) # grep BRCM_CC_4350 /usr/src/linux/drivers/net/wireless/brcm80211/brcmfmac/chip.c
case BRCM_CC_4350_CHIP_ID:

The resulting kernel module will be called brcmfmac, make sure to load it on boot by adding it in your /etc/conf.d/modules :

modules="brcmfmac"

EDIT: as of Jan. 7 2016, version 20151207 of linux-firmware ships with the needed files so you don’t need to download those any more !

Then we need to download the WiFi card’s firmware files which are not part of the linux-firmware package at the time of writing (20150012).

(chroot) # emerge '>=sys-kernel/linux-firmware-20151207'

# DO THIS ONLY IF YOU DONT HAVE >=sys-kernel/linux-firmware-20151207 available !
(chroot) # cd /lib/firmware/brcm/
(chroot) # wget http://ultrabug.fr/gentoo/xps9350/BCM-0a5c-6412.hcd
(chroot) # wget http://ultrabug.fr/gentoo/xps9350/brcmfmac4350-pcie.bin

Kernel config & build

I used genkernel to build my kernel. I’ve done a very few adjustments but these are the things to mind in this pre-built kernel :

  • support for NVME SSD added as builtin
  • it is builtin for ext4 only (other FS are not compiled in)
  • support for DM_CRYPT and LUKS ciphers for encrypted /home
  • the root partition is hardcoded in the kernel as /dev/nvme0n1p3 so if yours is different, you’ll need to change CONFIG_CMDLINE and compile it yourself
  • the CONFIG_CMDLINE above is needed because you can’t pass kernel parameters using UEFI so you have to hardcode them in the kernel itself
  • support for the intel graphic card DRM and framebuffer (there’s a kernel bug with skylake CPUs which will spam the logs but it still works good)

Get the kernel config and compile it :

(chroot) # mkdir -p /etc/kernels
(chroot) # cd /etc/kernels
(chroot) # wget http://ultrabug.fr/gentoo/xps9350/kernel-config-x86_64-4.3.3-gentoo
(chroot) # genkernel kernel

The proposed kernel config here is for gentoo-sources-4.3.3 so make sure to rename the file for your current version.

This kernel is far from perfect but it works very good so far, sound, webcam and suspend work smoothly !

make.conf settings for intel graphics

I can recommend using the following on your /etc/portage/make.conf :

INPUT_DRIVERS="evdev synaptics"
VIDEO_CARDS="intel i965"

fstab for SSD

Don’t forget to make sure the noatime option is used on your fstab for / and /home !

/dev/nvme0n1p1    /boot    vfat    noauto,noatime    1 2
/dev/nvme0n1p2    none     swap    sw                0 0
/dev/nvme0n1p3    /        ext4    noatime   0 1
/dev/nvme0n1p4    /home    ext4    noatime   0 1

As pointed out by stefantalpalaru on comments, it is recommended to schedule a SSD TRIM on your crontab once in a while, see Gentoo Wiki on SSD for more details.

encrypted /home auto-mounted at login

I advise adding the cryptsetup to your USE variable in /etc/portage/make.conf and then updating your @world with a emerge -NDuq @world.

I assume you don’t have created your user yet so your unmounted /home is empty. Make sure that :

  • your /dev/nvme0n1p4 home partition is not mounted
  • you removed the corresponding /home line from your /etc/fstab (we’ll configure pam_mount to get it auto-mounted on login)

AFAIK, the LUKS password you’ll set on the first slot when issuing luksFormat below should be the same as your user’s password !

(chroot) # cryptsetup luksFormat -s 512 /dev/nvme0n1p4
(chroot) # cryptsetup luksOpen /dev/nvme0n1p4 crypt_home
(chroot) # mkfs.ext4 /dev/mapper/crypt_home
(chroot) # mount /dev/mapper/crypt_home /home
(chroot) # useradd -m -G wheel,audio,video,plugdev,portage,users USERNAME
(chroot) # passwd USERNAME
(chroot) # umount /home
(chroot) # cryptsetup luksClose crypt_home

We’ll use sys-auth/pam_mount to manage the mounting of our /home partition when a user logs in successfully, so make sure you emerge pam_mount first, then configure the following files :

  • /etc/security/pam_mount.conf.xml (only line added is the volume one)
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE pam_mount SYSTEM "pam_mount.conf.xml.dtd">
<!--
	See pam_mount.conf(5) for a description.
-->

<pam_mount>

		<!-- debug should come before everything else,
		since this file is still processed in a single pass
		from top-to-bottom -->

<debug enable="0" />

		<!-- Volume definitions -->

<volume user="USERNAME" fstype="auto" path="/dev/nvme0n1p4" mountpoint="/home" options="fsck,noatime" />

		<!-- pam_mount parameters: General tunables -->

<!--
<luserconf name=".pam_mount.conf.xml" />
-->

<!-- Note that commenting out mntoptions will give you the defaults.
     You will need to explicitly initialize it with the empty string
     to reset the defaults to nothing. -->
<mntoptions allow="nosuid,nodev,loop,encryption,fsck,nonempty,allow_root,allow_other" />
<!--
<mntoptions deny="suid,dev" />
<mntoptions allow="*" />
<mntoptions deny="*" />
-->
<mntoptions require="nosuid,nodev" />

<!-- requires ofl from hxtools to be present -->
<logout wait="0" hup="no" term="no" kill="no" />


		<!-- pam_mount parameters: Volume-related -->

<mkmountpoint enable="1" remove="true" />


</pam_mount>
  • /etc/pam.d/system-auth (only lines added are the ones with pam_mount.so)
auth		required	pam_env.so 
auth		required	pam_unix.so try_first_pass likeauth nullok 
auth		optional	pam_mount.so
auth		optional	pam_permit.so

account		required	pam_unix.so 
account		optional	pam_permit.so

password	optional	pam_mount.so
password	required	pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3 
password	required	pam_unix.so try_first_pass use_authtok nullok sha512 shadow 
password	optional	pam_permit.so

session		optional	pam_mount.so
session		required	pam_limits.so 
session		required	pam_env.so 
session		required	pam_unix.so 
session		optional	pam_permit.so

That’s it, easy heh ?! When you login as your user, pam_mount will decrypt your home partition using your user’s password and mount it on /home !

UEFI booting your Gentoo Linux

The best (and weird ?) way I found for booting the installed Gentoo Linux and its kernel is to configure the UEFI boot directly from the XPS BIOS.

The idea is that the BIOS can read the files from the EFI boot partition since it is formatted as FAT32. All we have to do is create a new boot option from the BIOS and configure it to use the kernel file stored in the EFI boot partition.

  • reboot your machine
  • get on the BIOS (hit F2)
  • get on the General / Boot Sequence menu
  • click Add
  • set a name (like Gentoo 4.3.3) and find + select the kernel file (use the integrated file finder)

IMG_20160102_113759

  • remove all unwanted boot options

IMG_20160102_113619

  • save it and reboot

Your Gentoo kernel and OpenRC will be booting now !

Suggestions, corrections, enhancements ?

As I said, I wrote all this quickly to spare some time to whoever it could help. I’m sure there are a lot of improvements to be done still so I’ll surely update this article later on.

December 31, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
uWSGI v2.0.12 (December 31, 2015, 14:18 UTC)

It’s been a long time since I made a blog post about a uWSGI release but this one is special to me because it contains some features I asked for to a colleague of mine.

For his first contributions to a big Open Source project, our fellow @shir0kamii added two features (spooler_get_task and -if-hostname-match) which were backported in this release and that we needed at work for quite a long time : congratulations again :)

Highlights

  • official PHP 7 support
  • uwsgi.spooler_get_task API to easily read and inspect a spooler file from your code
  • -if-hostname-match regexp on uWSGI configuration files to allow a more flexible configuration based on the hostname of the machine

Of course, all of this is already available on Gentoo Linux !

Full changelog here as usual.

December 30, 2015
32C3 (December 30, 2015, 21:18 UTC)

This year I participated in the Chaos Computer Club's annual congress for the first time, despite it being the 32nd such event being arranged, hence its name 32c3. This year's event has the subname of "Gated Communities" and follows last year in its location in Hamburg after having been in Berlin for a while. By this point I expect many have written the event off as a nerd gathering of hackers, which, well, in many ways it is, but it requires some modification. The number of visitors exceeds 12,000, so this is a large event, lasting over four days from 27th to 30th of December each year, and if you look deeper into it actually is a family event for many with own events for teaching children technology and a childspace that include games that use technology to represent position or sound in order to control ping-pong-games. Picture taking is of course prohibited throughout the conference unless getting explicit permission from all involved parties (as it should be in the rest of society).

Presentations this year were organized in four main tracks, starting at 11:30 and going as late as 2am. It is a somewhat interesting experience to attend a lecture on "A gentle introduction to post-quantum cryptography" by Dan Bernstein and Tanja Lange at 23:00 - 00:00 and having a full lecture hall. I wonder how many universities would have the same result.

Don't worry though, if missing a lecture the video streaming is one of the better you can encounter, separated into multiple sections, (i) a live stream (ii) a Re-Live, which is un-modified version of the stream that can be watched later and (iii) A released video of the talk that is properly mastered and in better quality. So if wanting to watch the aforementioned talk on PQC you can do so at any time.

As a disproporational amount of my acquaintances are focusing on the legal field instead of technology in itself, lets continue with a good talk by Max Schrems suing Facebook over Safe Harbor and data protection going all the way to the european court of justice. Or maybe you want to learn more about the legal ambiguities surrounding Sealand, and the precesses involved in creation your own country and the operational failures of data havens?

If wanting to mix in the more technological part, how about a wrap-up of the Crypto Wars part II and comparisons to the 1990's. For those not having spent too much time looking into the first one, some particularly bad ideas were the clipper chip for key escrow, but what is curious is the same amount of arguments being used then as now. FBI/NSA and other governmental agencies wants un-fethered access to encrypted email and blames cryptography for its failures, even though those involved in recent events in Paris and San Bernadino actually used un-encrypted communication and the security services never picked up anything. As such, they, along with politicians, use Fear, Uncertainty, and Doubt (FUD) to make their case. Its typical of politicians to think that the problem is the rhethoric or the name rather than the underlying substance, and as a result we see discussions of a "secure golden key" or a "front door" instead of a "back door" to cryptography. The attempts of governments from the first crypto wars of couse influence us even today, in particular with the export restrictions imposed that until recently still exists compatibility for in various libraries allowing for downgrade attacks. A good talk by J. Alex Halderman and Nadia Heninger on Logjam underlines why attempts of undermining encryption is a bad thing even decades later.

What people seems to forget is that encryption is required for the e-commerce that we use every day. Who would ever connect to an internet banking application if their neighbour could be monitoring all account information and traffic? And the right to privacy is even established under the Universal Declaration of Human Rights, article 19, stating: "Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinion without interfearence and to seek, receive and impart information and ideas through any media and regardless of frontiers".

The United Kingdom (UK) is comming off this debate in a particularly bad way with Cameron's Snooper's Charter. In particular §189(4)(c): "Operators may be obliged to remove "electronic protection" if they provide ..." seems worrying. This is followed by Australia; where simply explaining an algorithm to someone can result in penalization. But none of these beats India; that require a copy of plain text to be retained for a minimum of 90 days if sending an encrypted message.

This level of tyranny from oppressive regimes of various governments nicely sets the stage for the presentation of North Korea's Red Star Operating System and the various ways the operating system, set to mimic Apple's Mac OS, in order to spy and keep the people down. Of particular interest is the watermarking technology and censoring application that forms part of the "anti-virus" (well, the red start icon of it could be a hint)

All in all, this is just a minimal representaiton of some of the interesting aspects of this conference. Not surprisingly the most used operating systems of the visitors (at least those connected to the network) was GNU/Linux (24.1%) and Android (17.6%), and if you want to see the talk about Windows 10 acting as a botnet you have that video as well.

December 28, 2015
Denis Dupeyron a.k.a. calchan (homepage, bugs)
SCALE 14x (December 28, 2015, 20:30 UTC)

I don’t know about you but I’m going to SCALE 14x.

I'm going to SCALE 14x!

SCALE is always a lot of fun and last year I particularly liked their new embedded track. This year’s schedule looks equally exciting. Among the speakers you may recognize former Gentoo developers as well as a few OSS celebrities.

We’ll have a booth again this year. Feel free to come talk to us or help us on the booth. One of the new things this year is we’re getting a Gentoo specific promotional code for a 50% rebate on full-access passes for the entire event. The code is GNTOO. It’s not only for Gentoo developers, but also for all Gentoo users, potential users, new-year-resolution future users, friends, etc… Use it! Spread it! I’m looking forward to see you all there.

December 19, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs)
XScreenSaver unlock dialog tuning (December 19, 2015, 19:42 UTC)

I’m having a bit of trouble accepting that one of the dialogs that is presented to me as frequently as the XScreenSaver unlock window below is by far the least shiny part of my daily Linux desktop experience.


Tuning just the knobs that XScreenSaver already comes with, I eventually got to this point:


The logo still is too much noise and the font still lacks anti-aliasing. However most of the text noise, the pre-90s aesthetics and the so-called thermometer are gone.

To bring it to your desktop, use this content for ~/.Xdefaults

xscreensaver.dateFormat:
xscreensaver.passwd.body.label:
xscreensaver.passwd.heading.label:
xscreensaver.passwd.login.label:
xscreensaver.passwd.thermometer.width: 2
xscreensaver.passwd.uname: False 
xscreensaver.passwd.unlock.label:

xscreensaver.Dialog.background:         #000000
xscreensaver.Dialog.foreground:         #ffffff
xscreensaver.Dialog.Button.background:  #000000
xscreensaver.Dialog.Button.foreground:  #ffffff
xscreensaver.Dialog.text.background:    #000000
xscreensaver.Dialog.text.foreground:    #ffffff

xscreensaver.Dialog.shadowThickness: 1
xscreensaver.Dialog.topShadowColor:     #000000
xscreensaver.Dialog.bottomShadowColor:  #000000

and run

xrdb < ~/.Xdefaults  && xscreensaver-command -restart

as advised by the XScreenSaver Manual.

For other approaches, I’m only aware of this one: xscreensaver lock window themes. Please comment below if you know about other approaches. Thank you!

PS: The screensaver in the background is Fireflies. For a Debian package, you can run make deb from a Git clone.

November 29, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
lxc, ipv6 and iproute2 (November 29, 2015, 21:19 UTC)

Not so recently I got a soyoustart system since it is provided with an option to install Gentoo out of box.

The machine comes with a single ipv4 and a /64 amount of ipv6 addresses.

LXC

I want to use the box to host some of my flask applications (plaid mainly), keep some continuous integration instances for libav and some other experiments with compilers and libraries (such as musl, cparser other).

Since Diego was telling me about lxc I picked it. It is simple, requires not much effort and in Gentoo we have at least some documentation.

Setting up

I followed the documentation provided and it worked quite well up to a point. The btrfs integration works as explained, creating new Gentoo instances just worked, setting up the network… Required some effort.

Network woes

I have just 1 single ipv4 and some ipv6 so why not leveraging them? I decided to partition my /64 and use some, configured the bridge to take ::::1::1 and set up the container configuration like this:

lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.ipv4 = 192.168.1.4/16
lxc.network.ipv4.gateway = auto
lxc.network.ipv6 = ::::1::4/80
lxc.network.ipv6.gateway = auto
lxc.network.hwaddr = 02:00:ee:cb:8a:04

But the route to my container wasn’t advertised.

Having no idea why I just kept poking around sysctl and iproute2 until I got:

  • sysctl.conf:
  net.ipv6.conf.all.forwarding = 1
  net.ipv6.conf.eth0.proxy_ndp = 1

And

ip -6 neigh add proxy ::::1::4 dev eth0

In my container runner script.

I know that at least other people had the problem so here this mini-post.

November 26, 2015
Andreas K. Hüttel a.k.a. dilfridge (homepage, bugs)
Grafting history onto your Gentoo git clone (November 26, 2015, 23:14 UTC)

Somehow after a while I got a bit tired that my git checkout of the main Gentoo repository didn't have any real history available. So, here's how I got it back:

(Note, you may want a fast network connection for this.)

  • cd into the main directory of the Gentoo git checkout:
$ cd ~/Gentoo/gentoo
  • fetch 2GByte of converted cvs history into a new local branch "history-20150809-draft"
$ git fetch https://github.com/gentoo/gentoo-gitmig-20150809-draft.git master:history-20150809-draft
  • attach the last commit of the cvs history to the first commit of the new git era
$ echo 56bd759df1d0c750a065b8c845e93d5dfa6b549d 2ebda5cd08db6bdf193adaa6de33239a83a73af0 > .git/info/grafts
And done. :)

Should at some point in the future a new, improved (or "official") conversion of the cvs history become available, here's (untested) what to do:
  • fetch it in the same way into a new local cvs history branch, and 
  • modify the grafts file to now connect the last commit of the new local cvs history branch with the first commit of the git era. 
Once you are happy with the result, you can delete the old local cvs history branch and run "git prune", freeing up the space used by the now obsolete old conversion.

Thanks to rich0 for providing the draft conversion (though inofficial so far) and to everyone else involved.

Alexys Jacob a.k.a. ultrabug (homepage, bugs)
py3status v2.7 (November 26, 2015, 18:52 UTC)

I’m more than two weeks late but I’m very glad to announce the release of py3status v2.7 which features a lot of interesting stuff !

For this release I want to salute the significant work and help of Daniel Foerster (@pydsigner), who discovered and fixed a bug in the event detection loop.

The result is a greatly improved click event detection and bar update speed with a largely reduced CPU consumption and less code !

Highlights

  • major performance and click event detection improvements by Daniel Foerster
  • support of %z on time and tztime modules fixes #110 and #123 thx to @derekdreery and @olhotak
  • directive %Z and any other failure in parsing the time and tztime modules format will result in using i3status date output
  • add ethernet, wireless and battery _first_ instance detection and support. thx to @rekoil for reporting on IRC
  • i3status.conf parser handles configuration values with the = char

New modules

  • new rt module: display ongoing tickets from RT queues
  • new xsel module: display xsel buffers, by umbsublime
  • new window_title_async module, by Anon1234

Modules enhancements

  • battery_level module: major improvements, documentation, add format option, by Maxim Baz
  • keyboard_layout module: color customisation, add format option, by Ali Mousavi
  • mpd_status module: fix connection leak, by Thomas Sanchez
  • pomodoro module: implement format option and add additional display features, by Christoph Schober
  • spotify module: fix support for new versions, by Jimmy Garpehäll
  • spotify module: add support for colors output based on the playback status, by Sondre Lefsaker
  • sysdata module: trim spaces in `cpu_temp`, by beetleman
  • whatismyip module: change default check URL and make it configurable

Thanks !

Once again, thanks to all contributors listed above !

November 21, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Code and Conduct (November 21, 2015, 18:54 UTC)

This is a sort of short list of checklists and few ramblings in the wake of Fosdem’s Code of Conduct discussions and the not exactly welcoming statements about how to perceive a Code of Conduct such as this one.

Code of Conduct and OpenSource projects

A Code of Conduct is generally considered a mean to get rid of problematic people (and thus avoid toxic situations). I prefer consider it a mean to welcome people and provide good guidelines to newcomers.

Communities without a code of conduct tend to reject the idea of having one, thinking that a is only needed to solve the above mentioned issue and adding more bureaucracy would just actually give more leeway to macchiavellian ploys.

That is usually a problem since, no matter how good things are now, it takes just few poisonous people to get in an unbearable situation and a you just need one in few selected cases.

If you consider the CoC a shackle or a stick to beat “bad guys” so you do not need it until you see a bad guy, that is naive and utterly wrong: you will end up writing something that excludes people due a quite understandable, but wrong, knee-jerk reaction.

A Code of Conduct should do exactly the opposite, it should embrace people and make easier joining and fit in. It should be the social equivalent of the developer handbook or the coding style guidelines.

As everybody can make a little effort and make sure to send code with spaces between operators everybody can make an effort and not use colorful language. Likewise as people would be more happy to contribute if the codebase they are hacking on is readable so they are more confident in joining the community if the environment is pleasant.

Making an useful Code of Conduct

The Code of Conduct should be a guideline for people that have no idea what the expected behavior, it should be written thinking on how to help people get along not on how to punish who do not like.

  • It should be short. It is pointless to enumerate ALL the possible way to make people uncomfortable, you are bound to miss it.
  • It should be understanding and inclusive. Always assume cultural bias and not ill will.
  • It should be enforced. It gets quite depressing when you have a 100+ lines code of conduct but then nobody cares about it and nobody really enforces it. And I’m not talking about having specifically designated people to enforce it. Your WHOLE community should agree on what is an acceptable behavior and act accordingly on breaches.

People joining the community should consider the Code of Conduct first as a request (and not a demand) to make an effort to get along with the others.

Pitfalls

Since I saw quite some long and convoluted wall of text being suggested as THE CODE OF CONDUCT everybody MUST ABIDE TO, here some suggestion on what NOT do.

  • It should not be a political statement: this is a strong cultural bias that would make potential contributors just stay away. No matter how good and great you think your ideas are, those unrelated to a project that should gather people that enjoy writing code in their spare time should stay away. The Open Source is already an ideology, overloading it with more is just a recipe for a disaster.
  • Do not try to make a long list of definitions, you just dilute the content and give even more ammo to lawyer-type arguers.
  • Do not think much about making draconian punishments, this is a community on internet, even nowadays nobody really knows if you are actually a dog or not, you cannot really enforce anything if the other party really wants to be a pest.

Good examples

Some CoC I consider good are obviously the ones used in the communities I belong to, Gentoo and Libav, they are really short and to the point.

Enforcing

As I said before no matter how well written a code of conduct is, the only way to really make it useful is if the community as whole helps new (and not so new) people to get along.

The rule of thumb “if somebody feels uncomfortable in a non-technical discussion, once he says, drop it immediately”, is ok as long:
* The person uncomfortable speaks up. If you are shy you might ask somebody else to speak up for you, but do not be quiet when it happens and then fill a complaint much later, that is NOT OK.
* The rule is not bent to derail technical discussions. See my post about reviews to at least avoid this pitfall.
* People agree to drop at least some of their cultural biases, otherwise it would end up like walking on eggshells every moment.

Letting situations going unchecked is probably the main issue, newcomers can think it is OK to behave in a certain way if people are behaving such way and nobody stops that, again, not just specific enforcers of some kind, everybody should behave and tell clearly to those not behaving that they are problematic.

Gentoo is a big community so once somebody steps the boundaries gets problematic having a swift reaction, lots of people prefer not to speak up when something happens, so people unwillingly causing the problem are not made aware immediately.

The people then in charge to dish bans have to try to figure out what exactly was wrong and there the cultural biases everybody has might or might not trigger and make the problem harder to address.

Libav is a much smaller community and in general nobody has qualms in saying “please stop” (that is also partially due how the community evolved).

Hopefully this post would help avoid making some mistakes and help people getting along better.

November 13, 2015
Michał Górny a.k.a. mgorny (homepage, bugs)
The Ultimate Guide to EAPI 6 (November 13, 2015, 21:13 UTC)

Now that EAPI 6 is Council-approved and pretty close to being deployed, I think it’s about time to write up a not-so-short guide to it. It’s especially important that EAPI 6 is a bit different from the previous EAPIs. It was not only seen as an opportunity to add new features but also to tidy things up a bit and improve strictness.

If you look into the PMS, you’d see that we’d not only added completely new stuff but also ported a few common eclass functions, in a little cleaner form, and new useful phase functions. Furthermore, we used EAPI 6 as an opportunity to finally increase strictness of Portage in regard to PMS, and ban some long-unwanted eclasses.

Therefore, I’d like to ask you — please don’t go hooray-enabling EAPI 6 support in your ebuilds and eclasses. Take a while to think, and do things right. Please think of EAPI 6 as a lifetime opportunity of improving your eclass APIs, as we improved the API provided by Package Managers.

Now, time for a little summary of changes, their implications and a bit of rationale.

Bash upgrade and compatibility setting

The first important change in EAPI 6 is the upgrade to bash-4.2 (the earlier EAPIs required bash-3.2). At a first glance, it just brings a few nice quirks like cleaner fd magic (used by multiprocessing.eclass), case conversion for variables and associative arrays. However, aside to that bash compatibility setting is forced — that could hopefully prevent existing code from breaking in future bash versions.

What does this solve, you ask? For example, it solves the tilde substitution issue:

PV="0_beta1"
MY_PV="${PV/_/~}"

The above snippet gives the expected result only when executed with bash-4.2 and lower. However, starting with bash-4.3 it performs tilde expansion on right-hand argument of the pattern substitution and puts your home directory in there. You can disable that via backslash-escaping the argument — but then it puts an extra backslash in older bash versions! How frustrating…

Now, if we have BASH_COMPAT fixed by the PM at 4.2, this discrepancy is no longer an issue. Because all supported versions of bash will force the old behavior, you can write the code without any conditionals, workarounds or worrying that it will suddenly break sometime in the future.

failglob in global scope

The second change to ebuild syntax is that filename expansion is now an explicit error in global scope. While this may sound scary at first, it wasn’t ever really supported. Note that this doesn’t affect phase functions, just the code executed when sourcing the ebuild. This is mostly targeted towards snippets like the following:

inherit python-r1

RDEPEND="$(python_gen_cond_dep 'dev-python/foo[${PYTHON_USEDEP}]' python2*)"

Do you see the problem here? python2* lacks quoting. This usually doesn’t cause any issues but if you happened to be in a directory with pythonFOOBAR file while sourcing the ebuild, it would suddenly pick it up and return pretty unexpected results. As a solution, an EAPI=6 ebuild would instantly abort here and require you to quote the pattern.

LC_COLLATE and LC_CTYPE settings

The description for locale settings may seem a little blurry but that’s for a reason. The original idea was that the Package Manager would export (pretty much like Paludis does):

LC_COLLATE=C
LC_CTYPE=C

to enforce stable sorting and case conversions. Without the former, some locales could cause results of globs such as patch files to apply in different order. Without the latter, some locales could give unexpected results of case changes. In particular, the Turkish tr_TR.UTF-8 locale is only too happy to map lowercase ‘i’ to uppercase ‘İ’ (yep, it has a dot above!)

So why is it worded this muddy? Because of our lovely Python and the lot of high quality scripts that are only too happy to interpret read files in the charset corresponding to the current locale, and therefore bail out hard when they don’t conform to the ASCII charset implied by the C locale.

So instead of C, we need a C.UTF-8 which hasn’t made its way into glibc for… how long exactly? Not to mention the portability. Therefore, we just request a seemingly sane locale, and hope we can work something out. In implementation, the Package Manager will most likely set LC_COLLATE=C directly and complain to the user if his LC_CTYPE does not conform.

Nevertheless, in EAPI 6 you should be able to safely remove your local LC_COLLATE and LC_CTYPE redefinitions, and hopefully also most of those excessive LC_ALL settings.

eapply, eapply_user and src_prepare()

This is probably the most controversial feature of the new EAPI. Those functions are intended as reasonably simple and predictable replacements for the commonly used eutils.eclass beasts, with new default src_prepare() to finish off the long-deprecated base.eclass.

eapply has a few substantial differences from epatch. However, I hope that people will nevertheless decide to abandon the latter, and use the new function, effectively also reducing the use of eutils.eclass. The differences are:

  • No -p* guessing. Instead, it defaults to -p1 and lets you override it. Now patches can apply predictably.
  • No magical configuration variables. All options are passed as parameters, followed by files. If options need to take separate arguments, -- is provided to separate options from files.
  • No compression support. There’s unpack that usually does that for you, you know, and patches in ${FILESDIR} are never compressed.
  • Simple directory handling. It just applies all *.diff and *.patch files from a directory. No custom suffixes, no exclusions.

While this may sound like a lot was lost, the function is still quite big and even twice as useful. Most importantly, all patches now apply (or fail) pretty predictably, and if they do fail, you get a clear output.

eapply_user provides user patch support on top of eapply. However, since user patches are a matter of configuration, the PMS really leaves the details to the Package Manager, and doesn’t even prohibit it from extending patching beyond eapply. eapply_user is considered obligatory, and can be safely called more than once (applying patches only the first time) to make eclass writing easier.

Both function are used in the default src_prepare() implementation which was inspired by the common implementations shared between multiple eclasses. In particular, it applies patches from the PATCHES variable (array strongly preferred due to ${FILESDIR}) and then user patches. Please note that in some cases, it would be advised to remove src_prepare() from your eclass altogether rather than calling the default from it.

einstalldocs and src_install()

This one’s much simpler and accepted in full agreement. It combines two categories of requests created against the documentation install code introduced in EAPI 4. Firstly, it splits the code off default_src_install() so that it can be reused without implicitly calling emake install. Secondly, it solves a number of problems with the original implementation.

So what’s the story here? EAPI 4 introduced a very nice src_install() phase which was able to install a few default documentation pieces automatically. That was quite liked, except that it was tightly bound with the default emake install call. As a result, a number of eclasses ended up (still) redefining it, with the most noticeable example of base.eclass having a split phase function for it. But since the eclass was long deprecated already, other eclasses had to copy the code rather than reuse it.

The idea of having a split function arose early during EAPI 6 work. While writing a new function, the issues and requests against the current default were collected and considered. Finally, the new function was given to early Council approval, and committed to eutils.eclass as a backport of the future EAPI 6 function. And as you can see now, it lands in EAPI 6 in exactly the same form, replacing the eutils implementation.

As a reminder, the changes from the original EAPI 4 implementation are:

  • empty DOCS variable disables installing documentation completely. Previously, it was undefined behavior (usually a dodoc usage error).
  • The DOCS variable can name directories as well as files.
  • HTML_DOCS variable has been added. It installs the listed files and/or directories (using dodoc, not deprecated dohtml!) into the html/ subdirectory of docdir.

As a result, the default src_install() implementation differs by the documentation install changes listed above. If you were overriding this phase just to get einstalldocs into it, you can remove it. If you were using einstalldocs from eutils.eclass, you do not have to inherit it anymore. And if you were inlining your own documentation install code, it is a good idea to replace it with standard one in EAPI 6.

get_libdir and econf changes

The next major target for EAPI 6 changes was the econf function, commonly used in src_configure(). Here, two goals were achieved. Firstly, a get_libdir function inspired by multilib.eclass was added. Secondly, --docdir and --htmldir parameters are now passed to configure scripts.

Why would get_libdir be related to econf at all? Well, except for the fact that it’s often used in configure script arguments, it pretty much shares the logic used to figure out libdir. So we’re not just moving a commonly used eclass function to the EAPI — we’re splitting the libdir determination logic off econf, and sharing it with ebuilds.

As for the two new parameters — --docdir and --htmldir — I don’t think they really need any comment here.

in_iuse

This one’s very controversial as well, and I’d personally prefer not having to introduce it at all. However, without it the developers would still resort to ugly hacks, so better to have a safe alternative. So, what’s the background?

Some developers found it quite convenient to do some conditionals in eclasses based on what ends up in IUSE. For example, if the ebuild has IUSE=static-libs, we magically enable code controlling static library build and install. Except for being ugly, this design had two major issues. Firstly, the code checking that was inconsistent and often buggy — some eclasses handled +static-libs, some didn’t. Secondly, the specification never mandated IUSE having any sane value throughout phase functions. So the results could have been quite surprising.

EAPI 6 solves both of those issues by providing a standard in_iuse function. All uses implementation-defined magic to check whether the specified flag is listed in the IUSE_EFFECTIVE conceptual variable. This includes not only flags set by ebuilds and eclasses but also implicitly added by profiles — in other words, any flag that is legal to the use function call.

I should note that in_iuse is legal only in phase functions — it is disallowed in global scope (i.e. while IUSE is still subject to changes). Nevertheless, I would heartily recommend avoiding it and using explicit conditional variables instead.

nonfatal die

Did you happen to dislike how some eclass-defined helpers could not be nonfatal properly? That’s no longer the case with EAPI 6. The die function has gained a simple -n option which makes it respect nonfatal, and therefore be a no-op when it is in effect.

In other words, you can write helpers like the following:

efoo() {
	#...
	foo || die -n "foo failed" || return ${?}
	#...
}

where die would simply return false and continue execution if nonfatal is in effect. Please remember that ‘nonfatal die’ does not return from the function implicitly. If you need to do so, you need to explicitly call return like in the above snippet.

unpack changes

Now we’re past the most interesting changes. For completeness, there were few small requests accepted for the unpack function:

  • Absolute and non-prefixed relative paths can be used now. The path handling is still a little creepy since pure filename (i.e. without a single slash) implicitly references ${DISTDIR} but anything with a slash is treated as a regular path.
  • .txz suffix is interpreted equivalently to .tar.xz. Previously, only the long form was accepted.
  • Suffix matching is case-insensitive now. Which means that .ZIP works as well as .zip. Hooray, we can now handle 8.3 filenames cleanly!

einstall banned, dohtml deprecated

Aside to new features, there are two important removals to be noted. The long-discouraged einstall function has been finally banned, and the dohtml function has been deprecated with the goal of removing it in EAPI 7.

The einstall command was intended as a cheap work-around for build systems that did not support DESTDIR= argument to emake install. Instead, it redefined a number of direct install paths (such as bindir=), passing the staging area path and target directory combined. Sadly, it has often been cause of confusion for new developers who have seen it as the standard install command (in line with econf, emake…).

Over a year ago, I’ve started tracking uses of einstall on behalf of QA, and noticed that most of the uses were completely unnecessary, either because the build system in question handled DESTDIR= correctly, or had a custom variable serving the same purpose. Furthermore, the function used parameter names matching autoconf while it was intended to deal with custom build systems which may require other variables. Therefore, it was decided that it is better to ban the command and inline the relevant make parameters whenever really unavoidable.

The dohtml function has been the official way of installing HTML documentation for quite a long time. No other command defined by PMS had as many options. Confusingly to some developers, it provided implicit filtering by file suffixes, with additional options to add additional suffixes, replace suffix list, filter by filename, exclude directories and directly alter install path. Moreover, there was yet another enhancement request asking for new suffixes in the default filter.

The Council has agreed with us that the function is overcomplex and confusing, and will be an object of never-ending requests and updates. Most of Gentoo developers have seen it as a way of installing files in the html/ subdirectory of docdir while it did so much more — and potentially confused them by excluding some files. It also resulted in absurd situations like people moving gtk-doc files out of the standard location used by gtk-doc just to match the ‘standard’ established by the function.

dohtml is officially deprecated since EAPI 6, and is expected to be banned in EAPI 7. The suggested replacement is dodoc which is much simpler, faster and in many cases will work the same (or even better, by restoring the files that were mistakenly missed by dohtml). It is possible that the current dohtml will eventually be moved to an eclass for the ebuilds that require filtering, yet the fate of this idea is unclear yet.

To help with the migration, Portage provides the following quirk (for make.conf or environment):

PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES=yes

When run with this, each dohtml invocation will verbosely list all files that were skipped through filtering. A null output indicates that dohtml can be safely replaced by dodoc.

Global-scope helpers banned in Portage

EAPI 6 brings not only the spec changes but also new strictness to Portage. In particular, almost all helper functions are now banned in global scope, as they are banned by PMS and other package managers. What does this imply? Most importantly, toolchain’s use of USE=multislot is now officially banned and will no longer work in EAPI 6.

Eclasses banned in EAPI 6

Finally, the changes brought by EAPI 6 will likely result in a number of API changes in multiple eclasses. Moreover, some of the existing eclasses will become completely unnecessary and will be banned.

The most important example here is the base.eclass which was semi-officially deprecated for a long time. In EAPI 6, it is fully replaced by default phase functions and must not be used. Similarly, I’ve decided to discontinue and ban autotools-utils.eclass and autotools-multilib.eclass, the former serving a similar purpose as base.eclass did in the past and the latter being just a wrapper atop multilib-minimal.eclass.

Aside to that, I’ve banned a number of eclasses that are deprecated currently, to avoid their use being carried on to the next EAPI.

Summary

To summarize all the changes in EAPI 6, let me assemble a check list of tasks that need to be done when updating ebuilds to EAPI 6:

  1. You can update your bash code to use bash-4.2 features, and remove unnecessary conditionals for future-incompatible behavior. It is unlikely that you have any.
  2. Ensure that all patterns that are supposed to be passed as-is to functions are quoted.
  3. Remove unnecessary LC_COLLATE and LC_CTYPE if they only reset it to the C (or POSIX) locale.
  4. If you are using epatch, try hard to replace it with eapply. You may need to pass an explicit -p value.
  5. If you are using epatch_user, you have to replace it with eapply_user. If you are not and you define src_prepare(), you have to add it there.
  6. If you are left with src_prepare() that looks pretty much like the default one, drop it. It is also a good idea to remove unnecessary src_prepare() definitions from eclasses.
  7. If you are using einstalldocs, this function is now part of EAPI. Don’t count it as a reason to inherit eutils.eclass. If you are using a similar inline code, it’s about time you switch to einstalldocs.
  8. If you are using in_iuse, this function is now part of EAPI and even really legal. Don’t count it as a reason to inherit eutils.eclass. In fact, hurry to switch to EAPI 6 to get a predictable behavior!
  9. Check if you are still using anything from eutils.eclass. If you are not, remove the inherit and save a few cycles.
  10. If you are using get_libdir, this function is now part of EAPI. Don’t count it as a reason to inherit multilib.eclass. If you are not using anything else from multilib.eclass, remove the inherit.
  11. If you are passing standard --docdir and/or --htmldir to econf, you can remove them. EAPI 6 does that for you.
  12. If you are defining ebuild helpers that should respect nonfatal, you should update appropriate die to use die -n || return. However, please note that it’s usually a good idea to use regular die on usage errors, and respect nonfatal only for external errors.
  13. If you are using hacks to support .txz, uppercase archive suffixes like .ZIP or extracting files by absolute or relative paths, you can remove them and use new unpack features.
  14. If you are using einstall, remove it. I’m pretty sure you never really needed it. And if you really did, then you most likely missed some custom prefix option in upstream build system.
  15. If you are using dohtml, consider replacing it with dodoc -r. You can set PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES=yes before doing that, to see if the two will be equivalent.
  16. If you are inheriting base.eclass, autotools-utils.eclass or any of the other banned eclasses, make sure to remove it from your eclasses and ebuilds.

November 06, 2015
Luca Barbato a.k.a. lu_zero (homepage, bugs)
Reviews (November 06, 2015, 17:03 UTC)

This spurred from some events happening in Gentoo, since with the move to git we eventually have more reviews and obviously comments over patches can be acceptable (and accepted) depending on a number of factors.

This short post is about communicating effectively.

When reviewing patches

No point in pepper coating

Do not disparage code or, even worse, people. There is no point in being insulting, you add noise to the signal:

You are a moron! This is shit has no place here, do not do again something this stupid.

This is not OK: most people will focus on the insult and the technical argument will be totally lost.

Keep in mind that you want people doing stuff for the project not run away crying.

No point in sugar coating

Do not downplay stupid mistakes that would crash your application (or wipe an operating system) because you think it would hurt the feelings of the contributor.

    rm -fR /usr /local/foo

Is as silly as you like but the impact is HUGE.

This is a tiny mistake, you should not do that again.

No, it isn’t tiny it is quite a problem.

Mistakes happen, the review is there to avoid them hitting people, but a modicum of care is needed:
wasting other people’s time is still bad.

Point the mistake directly by quoting the line

And use at most 2-3 lines to explain why it is a problem.
If you can’t better if you fix that part yourself or move the discussion on a more direct media e.g. IRC.

Be specific

This kind of change is not portable, obscures the code and does not fix the overflow issue at hand:
The expression as whole could still overflow.

Hopefully even the most busy person juggling over 5 different tasks will get it.

Be direct

Do not suggest the use of those non-portable functions again anyway.

No room for interpretation, do not do that.

Avoid clashes

If you and another reviewer disagree, move the discussion on another media, there is NO point in spamming
the review system with countless comments.

When receiving reviews (or waiting for them)

Everybody makes mistakes

YOU included, if the reviewer (or more than one) tells you that your changes are not right, there are good odds you are wrong.

Conversely, the reviewer can make mistakes. Usually is better to move away from the review system and discuss over emails or IRC.

Be nice

There is no point in being confrontational. If you think the reviewer is making a mistake, politely point it out.

If the reviewer is not nice, do not use the same tone to fit in. Even more if you do not like that kind of tone to begin with.

Wait before answering

Do not update your patch or write a reply as soon as you get a notification of a review, more changes might be needed and maybe other reviewers have additional opinions.

Be patient

If a patch is unanswered, ping it maybe once a week, possibly rebasing it if the world changed meanwhile.

Keep in mind that most of your interaction is with other people volunteering their free time and not getting anything out of it as well, sometimes the real-life takes priority =)

November 05, 2015
Mike Pagano a.k.a. mpagano (homepage, bugs)
kdbus not in gentoo-sources (November 05, 2015, 23:33 UTC)

As per upstream developers instructions to Fedora maintainers [1] , kdbus will not be included in any new gentoo-sources genpatches release.

Of course, I will revisit when upstream feels it’s ready to be tested in distributions once again.

[1] http://lwn.net/Articles/663062/

October 20, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
uhashring : consistent hashing in python (October 20, 2015, 21:05 UTC)

It’s been quite some time since I wanted to use a consistent hashing based distribution of my workload in a quite big cluster at work. So when we finally reached the point where this became critical for some job processing I rushed to find out what python library I could use to implement this easily and efficiently.

I was surprised not to find a clear “winner” for such a library. The more “popular” named hash_ring has a rather unoptimized source code and is dead (old issues, no answer). Some others stepped up since but with no clear interest for contributions and almost no added features for real world applications.

So I packaged the ketama C library and its python binding on my overlay to get intimate with its algorithm. Then I started working on my own pure python library and released uhashring on Pypi and on Gentoo portage !

Features

  • instance-oriented usage so you can use your consistent hash ring object directly in your code (see advanced usage).
  • a lot of convenient methods to use your consistent hash ring in real world applications.
  • simple integration with other libs such as memcache through monkey patching.
  • all the missing functions in the libketama C python binding (which is not even available on pypi).
  • another and more performant consistent hash algorithm if you don’t care about the ketama compatibility (see benchmark).
  • native pypy support.
  • tests of implementation, key distribution and ketama compatibility.

WTF ?

Not so sure about hash tables and consistent hashing ? Read my consistent hashing 101 explanation attempt !

October 09, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Important!

My tech articles—especially Linux ones—are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Yet again, I had troubles with restarting amavisd after some package updates. You can see about the other problems that I’ve had with the application in these two previous blog posts: missing SpamAssassin rules causing amavis to fail and start-up failures due to a missing Perl IPv6 module.

This time, the problem was related to the backend database used for Bayes filtering within SpamAssassin, although it wasn’t readily clear what the problem was. After updating SpamAssassin, I saw the following message from Portage (Gentoo‘s package manager):


* You need to configure your database to be able to use Bayes filter
* with database backend, otherwise it will still use (and need) the
* Berkeley DB support.
* Look at the sql/README.bayes file in the documentation directory
* for how to configure it.

At first, I thought that there was a problem with my MariaDB installation, but then I remembered that I’m currently still using BerkeleyDB (BDB) for SpamAssassin.

After trying to stop and restart amavisd, zapping the configuration (which was a problem mentioned in one of my previous posts), manually killing the running amavisd processes, and a few other things, I was still left with a configuration that would cause amavisd to fail on startup:


* Starting amavisd-new ...
Problem in Amavis::DB or Amavis::DB::SNMP code:
BerkeleyDB needs compatible versions of libdb & db.h
you have db.h version 6.0.30 and libdb version 6.0.35
Compilation failed in require at (eval 93) line 20.
BEGIN failed--compilation aborted at (eval 93) line 20. [ !! ]
* ERROR: amavisd failed to start

Out of sheer desperation, I attempted what seemed to be an unlikely fix by recompiling BerkleyDB (via emerge -v dev-perl/BerkeleyDB). Lo and behold, amavis decided to start after that recompilation! It’s a bit strange to me that the standard tools like revdep-rebuild and emerge @preserved-rebuild didn’t catch this problem.

In any case, third time is a charm with amavisd. Hopefully there won’t be any further problems when attempting to reload or restart it.

Cheers,
Zach

October 04, 2015
Sebastian Pipping a.k.a. sping (homepage, bugs)

Hi!

Frequent updates are important, especially with Gentoo. You probably have seen conflicts that filled several screens and were not trivial to resolve. With frequent updates, handling conflicts becomes a lot easier. Frequent updates also gets you security fixes earlier. I would have done it from the beginning, if something would have reminded me to update. By now, for notifications on available updates I use

Former is a tool by Benedikt Böhm sending e-mails with emerge --pretend output whenever it changes. Latter is a bit of a quick hack by me that sits in the tray and shows an icon if updates are available with the number of updates in a tooltip. It requires a cron job to run something like eix-sync once a day but that’s probably a good idea anyway.
Maybe it’s something that you want to use too. If you run into bugs, please report them and share your patches if any. GitHub allows attaching text files by now, it may work with patches, too.

I would be curious to hear what other tools you use for being notified of updates on Gentoo. Please leave a comment below.

Best, Sebastian

Latest Toy: The Orange Pi PC (October 04, 2015, 14:33 UTC)

You may have heard about the latest Raspberry Pi 2 clone: The Orange Pi PC, a device that sells for only $15 yet comes with a quad-core Allwinner H3 and 1 GB RAM. (But don't be fooled - while "open source" is mentioned everywhere on that website, the actual open source situation is still pretty bad).


After my Raspberry Pi started acting up and run very unstable a year ago, this looked like a nice replacement. Thus I decided to order the "Orange Pi PC set 3" including a plastic case and power cable from AliExpress, and will try to get Gentoo running on that device. The manufacturer's Gentoo forum section is all crickets, so I expect to encounter at least some problems.

If you want to see this thing in action, or at least watch me struggling to get Gentoo running, come next weekend (10+11 October 2015) to the Prague Linux Days and visit the Gentoo booth.

October 02, 2015
Rafael Goncalves Martins a.k.a. rafaelmartins (homepage, bugs)
blogc: helper tools (October 02, 2015, 03:51 UTC)

While users may be able to use blogc as is with the help of generic tools, some of these tools are really boring to setup.

With that in mind, I'm trying to develop some simple tools to help the users getting their blogs online. At this point I have two tools ready for usage:

Packages are available for Gentoo, and in my Copr repository for RHEL/CentOS/Fedora.

blogc-git-receiver

blogc-git-receiver is a login shell and a git pre-receive hook, that can create git repositories and build/deploy your websites automatically. Users just need to create an user, configure it to use blogc-git-receiver as its login shell, then every time that some authorized user pushes to a repository it will create a local bare repository in the server, if needed, and if the push includes some change to the master branch, it will rebuild your website for you.

blogc-git-receiver tries to be as atomic as possible, building the new version of the website in a separate directory, and using a symlink to point to the most recent version of the website, removing the old version only after a successful new build.

blogc-git-receiver creates a symlink inside the bare git repository, called htdocs, that points to the last successful build. Users just need to make their webservers use this symlink as document root for their virtual host, and make sure that the webserver can follow symlinks.

With this tool, users can create their own PaaS-like environment, using a cheap VPS to host lots of blogs. ;-)

This tool is one of the reasons why I wanted to make blogc as fast as possible, because it will rebuild all the website every time, not just the changes, for the sake of consistency.

This tool is also a good sample code for people interested in understand how a login shell and a git hook works.

Gentoo package is dev-vcs/blogc-git-receiver and RHEL/CentOS/Fedora package is blogc-git-receiver.

Some simple documentation is available at: https://github.com/blogc/blogc-git-receiver

blogc-runserver

blogc-runserver is a simple HTTP server, that comes with several rules pre-defined, that tries to mimic the way most production servers work when serving static websites. Users just need to point blogc-runserver to the output directory where blogc created its result files.

A simple Makefile rule is able to run your website for testing:

serve: all
	blogc-runserver $(OUTPUT_DIR)

Yeah, it is really that simple! :)

Please keep in mind that this server should not be used in production. It is really simple, and developed for testing purposes.

Gentoo package is www-servers/blogc-runserver and RHEL/CentOS/Fedora package is blogc-runserver.

Some simple documentation is available at: https://github.com/blogc/blogc-runserver

Other tools

I have more ideas of new tools, that I will probably explain in future posts, but if you have ideas of useful tools, please let me know.

Thanks.

September 29, 2015
Paweł Hajdan, Jr. a.k.a. phajdan.jr (homepage, bugs)
Gentoo's top-notch user community (September 29, 2015, 19:39 UTC)

One of the best parts of Gentoo for me is the community.

For example, I regularly receive email from people who volunteer to help with testing hard masked www-client/chromium ebuilds. FWIW you don't need to email me or the Gentoo Chromium team - just start testing and filing bugs. On the other hand, I do appreciate when people express interest in helping out.

Another example is helping with getting bugs resolved.

Bug #556812 looked rather mysterious - until Guillaume ZITTA found that "build" is a red herring, and in fact the "tracing" module being imported is a different one (from dev-python/tracing as opposed to chromium sources). It was an unfortunate names collision - once found, quite easy to fix.

In bug #553502 Johan Hovold found we need to require postproc USE flag for libvpx to avoid a crash.

See bug #551666 for some Alexey Dobriyan's gdb magic mostly based on a single line segfault report from dmesg...

These are just a few examples.

By the way, the area where we could use more help is arm architecture support. Some specific bugs where help is wanted are #555702#545368#520180, and #483130. Please report whether you can reproduce or not, post emerge --info from your system and the compressed build log in case build fails.

September 27, 2015
Domen Kožar a.k.a. domen (homepage, bugs)
Friends sometimes let friends curl to shell (September 27, 2015, 17:50 UTC)

Every now and then (actually quite often), people complain on twitter they're afraid of our simple bash installer for Nix package manager:

$ bash <(curl https://nixos.org/nix/install)

Example (from today):

There are popular blog posts discouraging use of it.

Ask yourself a question, how would package manager install itself? Via another package manager?

If we assume nixos.org is not compromised (which is really hard to detect), using TLS to secure connection and with our simple trick to prevent partial download execution (you haven't read the script yet, right?), what can really go wrong?

It's the most transparent way to see how the package manager can be bootstrapped: read the source, Luke.

If you still have a reason why piping to shell is a bad idea, let me know.

September 24, 2015
Matthew Thode a.k.a. prometheanfire (homepage, bugs)
OpenStack on Gentoo is awesome (September 24, 2015, 05:00 UTC)

Some may wonder why to run OpenStack on Gentoo, it's akin to running one extremely complex piece of software on another potentially extremely complex operating system.

I propose Gentoo as the most correct operating system to run OpenStack as Gentoo is best prepared to handle some of the complexities that come with OpenStack.

Things Gentoo does well for OpenStack

  • Dependency resolution - we can take upstream's dependencies (requirements.txt) and map it directly to Gentoo's packages without changing the versions supported. This is not always the case in other distributions.
  • Because of this we can better rely on upstream's testing as validation that OpenStack on Gentoo will work properly.
  • useflags - some dependencies (such as memcached, qemu, etc) are optional depending on what services you are running.
  • python3 support - as upstream is finally moving to python3 we are easily able to extend support as it occurs.
  • patching - https://wiki.gentoo.org/wiki//etc/portage/patches allows users to patch anything as needed.
  • Upstream vanilla Openstack - We are not carrying many (I can count them on one hand) patches that are not upstreamed.
  • Testing - All the major services have testing support to validate good installs and / or custom patches not breaking things.

Future of OpenStack on Gentoo

  1. The Liberty release 15/10/2015
    • Upstream is reversioning the major components of OpenStack, 2015.2.0 will not exist, it will be something like 12.0.0.
    • The reversioning will mean some manual intervention if you wish to use Liberty on Gentoo, namely you will need to mask greater than version 2000 of the major components of OpenStack
  2. More services
    • I will at least add Heat during the next (Liberty) release cycle, possibly more
    • I will investigate readding Horizon, but doubt it'll be easily packagable.
  3. Security
    • I'd like to look into some of the selinux policies that are being developed for some of the services. I know nova/qemu has apparmor support.

Limits of testing

You cannot directly test OpenStack packages by emerging with USE="test" FEATURES="test" because there are interdepenecies that cause loops in the depgraph of portage for the (test) dependencies. You can get around it one way though.

# install only the test deps
emerge --onlydeps --oneshot --with-test-deps sys-cluster/nova
# test and install the actual package
USE="test" FEATURES="test" emerge sys-cluster/nova

September 23, 2015
Alex Legler a.k.a. a3li (homepage, bugs)
Repackaging packages.gentoo.org (September 23, 2015, 18:59 UTC)

Update: The site is now live at packages.gentoo.org!

Gentoo has seen quite some refreshing updates this year: In April, our website was relaunched (twice ;)) and our main package repository is now finally powered by git. There is one service though that sort-of relates to both, but this service has not seen an update in quite some time: Let’s talk about packages.gentoo.org.

The fact that we now use git for what was before called gentoo-x86  requires a set of changes to our package website. Several repository URLs are different now, and especially the changelogs that are now taken from git log messages need new code to display properly. As many participants in our website survey a while back have also noted, the time is most ripe for a change of scenery on “pgo”.

As having a good package search is one of the most-requested feature for the site, I’ve built a new site around elasticsearch (which also powers the new Gentoo mailing list archives). It is nearing completion as a minimum viable product, but already available for you to test in a beta state.

So, head over to https://packages.gentoo.org/ https://packagestest.gentoo.org/ and feel free to browse around. Some things are not available yet and the caching has not been fully enabled, so things might take a second to render. I’d be glad to hear your thoughts and ideas on the site—either as a comment here, or if you prefer on freenode in #gentoo-www.

As a sneak peek, here’s the site rendered on a small screen in case you don’t have your $device handy:

September 20, 2015
Rafael Goncalves Martins a.k.a. rafaelmartins (homepage, bugs)
blogc, a blog compiler (September 20, 2015, 13:20 UTC)

So, this is a follow-up for my previous post, about my "new blog".

I promissed to write something about the "new blogging engine" that runs it, and my plan was to finish some documentation and release a first "stable" version of the project before writing this post. And surprisingly it is ready!

After a few beta versions, mainly to verify build system and packaging, I released the first "stable" version a few days ago, with some simple documentation (man pages) and all the features required for basic usage.

This blog is powered by blogc, that is a blog compiler. It works like an usual source code compiler, converting input files written in a custom input language into a target file, based on a template. It can generate almost any resource file needed by a website/blog as target file, depending on the template used, like posts, pages, feeds, sitemaps, etc.

Hmm, but this isn't similar to what [put your favorite static site generator here] does?!

Yeah, it is, indeed. But blogc was developed to be:

  1. Stateless and idempotent: no global state is shared by the binary between compiler calls, and compiler calls with the same arguments should always generate the same output. The compiler should not need to manage what needs to be rebuilt or not, this is a job for a build tool like make.
  2. Fast: the compiler should be able to generate a complex target file in a few miliseconds, making it easy to regenerate the full website/blog whenever needed. This is a feature required by another component of this project, that I'll introduce in another post.
  3. Simple: Both the source and template syntaxes are simple enough to be learned quickly, and are specific for this use case.
  4. Easy to host: it produces static files, that can be hosted anywhere.

As most of the people who read this blog knows, I like to research and implement stuff related to personal content publishing, despite not being a frequent blogger. This time I wanted to put in practice some concepts I had in mind for some time, and play with handwritten parsers.

This project is built around 3 parsers, that are basically state machines. The parsers are heavily inspired by nodejs' http parser, that is inspired by nginx's http-parser. This "methodology" of building parsers make it easy to create fast and memory efficient parsers.

blogc is totally implemented in C99, without any external dependencies.

blogc comes with a somewhat complete unit test suite, that depends on cmocka. Obviously, users who don't want to run the test suite don't need to install cmocka.

I'll provide more implementation details in a future post.

OK, how can I install it?! :D

Gentoo users can install blogc from the main package tree. Please note that the package is keyworded for ~x86 and ~amd64:

# emerge -av app-text/blogc

Fedora users, and users of other Linux distributions that support the EPEL repository, can install it from my Copr repository.

After enabling this repository, just type:

# yum install blogc

I have plans to package it for Ubuntu/Debian, but this is low priority right now.

Users without access to prebuilt packages can install blogc from source. Just visit https://blogc.rgm.io, download a source tarball, extract it and build. It is an autotools project, that builds with the usual command combo:

$ ./configure
$ make
# make install

And how can I get started with blogc?

The website contains lots of useful information.

With blogc installed, you can read man pages: blogc(1), blogc-source(7) and blogc-template(7). They are also available in the website.

You can also use our example website as base for your own website. It is very simple, but should be a good start point:

$ git clone https://github.com/blogc/blogc-example.git my-blog
$ cd my-blog
$ rm -rf .git
$ git init .
$ git add .
$ git commit -m 'My new blog, initialized from blogc-example'

The example website uses a Makefile to build the website. Reading this Makefile is a good way to learn about how blogc works. This Makefile just works with GNU Make, but users should be able to write a Makefile that works with any Make implemantation that follows POSIX standards, if wanted. Users can also use task runners like Grunt, if they know how to make it call blogc properly. :-)

What the name "blogc" means?

"blogc" stands for "blog compiler". The name has nothing to do with the programming language used to implement it. ;-)

And what about blohg? It was abandoned?

I'm still maintaining it, and it seems to be stable enough for general usage at this point. Feel free to contact me if you find some bug on it.

That's it for now.

I hope you like blogc!

September 14, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)

In my last post, I explained the new hope we had in following some newly added recommended steps before trying to migrate our production cluster to mongoDB 3.0 WiredTiger.

The most demanding step was migrating all our production servers data storage filesystems to XFS which obviously required a resync of each node… But we ended up being there pretty fast and were about to try again as 3.0.5 was getting ready, until we saw this bug coming !

I guess you can understand why we decided to wait for 3.0.6… which eventually got released with a more peaceful changelog this time.

The 3.0.6 crash test

We decided as usual to test the migration to WiredTiger in two phases.

  1. Migrate all our secondaries to the WiredTiger engine (full resync). Wait a week to see if this has any effect on our cluster.
  2. Switch all the MMapv1 primary nodes to secondary and let our WiredTiger secondary nodes become the primary nodes of our cluster. Pray hard that this time it will not break under our workload.

Step 1 results were good, nothing major changed and even our mongo dumps were still working this time (yay!). One week later, everything was still working smoothly.

Step 2 was the big challenge which failed horribly last time. Needless to say that we were quite stressed when doing the switch. But it worked smoothly and nothing broke + performances gains were huge !

The results

Nothing speaks better than metrics, so I’ll just comment them quickly as they speak by themselves. I obviously can’t disclose the scales sorry.

Insert-only operations gained 25x performance

2015-09-14-173545_531x276_scrot

 

 

 

 

 

 

 

Upsert-heavy operations gained 5x performance

2015-09-14-173748_530x275_scrot

 

 

 

 

 

 

 

Disk I/O also showed mercy to the disk overall usage. This is due to WiredTiger superior caching and disk flushing mechanisms.

2015-09-14-173954_409x254_scrot

 

 

 

 

 

 

Disk usage decreased dramatically thanks to WiredTiger compression

2015-09-14-174137_317x180_scrot

 

 

 

 

The last and next step

As of today, we still run our secondaries with the MMapv1 engine and are waiting a few weeks to see if anything goes wrong in the long run. Shall we need to roll back, we’d be able to do so very easily.

Then when we get enough uptime using WiredTiger, we will make the final switch to a full Roarring production cluster !

September 10, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Custom CIL SELinux policies in Gentoo (September 10, 2015, 05:13 UTC)

In Gentoo, we have been supporting custom policy packages for a while now. Unlike most other distributions, which focus on binary packages, Gentoo has always supported source-based packages as default (although binary packages are supported as well).

A recent commit now also allows CIL files to be used.

Policy ebuilds, how they work

Gentoo provides its own SELinux policy, based on the reference policy, and provides per-module ebuilds (packages). For instance, the SELinux policy for the screen package is provided by the sec-policy/selinux-screen package.

The package itself is pretty straight forward:

# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id$
EAPI="5"

IUSE=""
MODS="screen"

inherit selinux-policy-2

DESCRIPTION="SELinux policy for screen"

if [[ $PV == 9999* ]] ; then
        KEYWORDS=""
else
        KEYWORDS="~amd64 ~x86"
fi

The real workhorse lays within a Gentoo eclass, something that can be seen as a library for ebuilds. It allows consolidation of functions and activities so that a large set of ebuilds can be simplified. The more ebuilds are standardized, the more development can be put inside an eclass instead of in the ebuilds. As a result, some ebuilds are extremely simple, and the SELinux policy ebuilds are a good example of this.

The eclass for SELinux policy ebuilds is called selinux-policy-2.eclass and holds a number of functionalities. One of these (the one we focus on right now) is to support custom SELinux policy modules.

Custom SELinux policy ebuilds

Whenever a user has a SELinux policy that is not part of the Gentoo policy repository, then the user might want to provide these policies through packages still. This has the advantage that Portage (or whatever package manager is used) is aware of the policies on the system, and proper dependencies can be built in.

To use a custom policy, the user needs to create an ebuild which informs the eclass not only about the module name (through the MODS variable) but also about the policy files themselves. These files are put in the files/ location of the ebuild, and referred to through the POLICY_FILES variable:

# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id$
EAPI="5"

IUSE=""
MODS="oracle"
POLICY_FILES="oracle.te oracle.if oracle.fc"

inherit selinux-policy-2

DESCRIPTION="SELinux policy for screen"

if [[ $PV == 9999* ]] ; then
        KEYWORDS=""
else
        KEYWORDS="~amd64 ~x86"
fi

The eclass generally will try to build the policies, converting them into .pp files. With CIL, this is no longer needed. Instead, what we do is copy the .cil files straight into the location where we place the .pp files.

From that point onwards, managing the .cil files is similar to .pp files. They are loaded with semodule -i and unloaded with semodule -r when needed.

Enabling CIL in our ebuilds is a small improvement (after the heavy workload to support the 2.4 userspace) which allows Gentoo to stay ahead in the SELinux world.

September 07, 2015
Bernard Cafarelli a.k.a. voyageur (homepage, bugs)
Testing clang 3.7.0 OpenMP support (September 07, 2015, 15:57 UTC)

Soon after llvm 3.7.0 reached the Gentoo tree, Jeremi Piotrowski opened two bugs to fix OpenMP support in our ebuilds. sys-devel/llvm-3.7.0-r1 (a revision bump that also now installs LLVM utilities) now has a post-install message with a short recap, but here is the blog version.

As detailed in upstream release notes, OpenMP in clang is disabled by default (and I kept this default in the Gentoo package), and the runtime is a separate package. So:

  • emerge sys-libs/libomp
  • add “-fopenmp=libomp” in your CFLAGS
  • check with a sample program like:
  • #include <stdio.h>
    #include <omp.h>
    
    int main() {
    #pragma omp parallel
        printf("Hello from thread %d, nthreads %d\n", omp_get_thread_num(), omp_get_num_threads());
    }
  • You should get multiple lines on STDOUT (default threads count is one per CPU):
  • % clang -fopenmp=libomp test_omp.c -o test_omp
    % OMP_NUM_THREADS=6 ./test_omp
    Hello from thread 4, nthreads 6
    Hello from thread 5, nthreads 6
    Hello from thread 1, nthreads 6
    Hello from thread 3, nthreads 6
    Hello from thread 0, nthreads 6
    Hello from thread 2, nthreads 6

Some additional reading:

September 06, 2015
Patrick Lauer a.k.a. bonsaikitten (homepage, bugs)
Printers in Linux ... (September 06, 2015, 11:46 UTC)

I was trying to make a Brother printer do some printing. (Optimistic, I know ... but ...)
Out of the box CUPS can't handle it. Since it's CUPS it doesn't do errors, so who knows why. Anyway, the autoconfigured stuff ends up being eh whut ur doin is naught gud.

The config file points at a hostname that is not even within our local network, so I think this won't print locally. Cloud printing? Cups can do!
There's a printer driver from Brother, which is horribly stupid, bad, stupid, and bad. I read through the 'installer' script trying to figure out what it would do, but then I realized that I'm not on an RPM distro so it's futile.

So then I figured "why not postscript?"

And guess what. All the documentation I found was needlessly wrong and broken, when all you have to do is configure an IPP printer, generic postscript, and ... here comes instant good quality colour duplex output from the printer.

I'm confused about many things here - the complexity of Brother's attempt at a driver that doesn't work, CUPS autoconfiguration that sees the printer and then derps out, and all the documentation that doesn't point out that all this weird layercake of workarounds is not needed because it's a standards-compliant postscript printer.
How exhausting.

September 02, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Maintaining packages and backporting (September 02, 2015, 18:33 UTC)

A few days ago I committed a small update to policycoreutils, a SELinux related package that provides most of the management utilities for SELinux systems. The fix was to get two patches (which are committed upstream) into the existing release so that our users can benefit from the fixed issues without having to wait for a new release.

Getting the patches

To capture the patches, I used git together with the commit id:

~$ git format-patch -n -1 73b7ff41
0001-Only-invoke-RPM-on-RPM-enabled-Linux-distributions.patch
~$ git format-patch -n -1 4fbc6623
0001-Set-self.sename-to-sename-after-calling-semanage_seu.patch

The two generated patch files contain all information about the commit. Thanks to the epatch support in the eutils.eclass, these patch files are immediately usable within Gentoo's ebuilds.

Updating the ebuilds

The SELinux userspace ebuilds in Gentoo all have live ebuilds available which are immediately usable for releases. The idea with those live ebuilds is that we can simply copy them and commit in order to make a new release.

So, in case of the patch backporting, the necessary patch files are first moved into the files/ subdirectory of the package. Then, the live ebuild is updated to use the new patches:

@@ -88,6 +85,8 @@ src_prepare() {
                epatch "${FILESDIR}/0070-remove-symlink-attempt-fails-with-gentoo-sandbox-approach.patch"
                epatch "${FILESDIR}/0110-build-mcstrans-bug-472912.patch"
                epatch "${FILESDIR}/0120-build-failure-for-mcscolor-for-CONTEXT__CONTAINS.patch"
+               epatch "${FILESDIR}/0130-Only-invoke-RPM-on-RPM-enabled-Linux-distributions-bug-534682.patch"
+               epatch "${FILESDIR}/0140-Set-self.sename-to-sename-after-calling-semanage-bug-557370.patch"
        fi

        # rlpkg is more useful than fixfiles

The patches themselves do not apply for the live ebuilds themselves (they are ignored there) as we want the live ebuilds to be as close to the upstream project as possible. But because the ebuilds are immediately usable for releases, we add the necessary information there first.

Next, the new release is created:

~$ cp policycoreutils-9999.ebuild policycoreutils-2.4-r2.ebuild

Testing the changes

The new release is then tested. I have a couple of scripts that I use for automated testing. So first I update these scripts to also try out the functionality that was failing before. On existing systems, these tests should fail:

Running task semanage (Various semanage related operations).
  ...
    Executing step "perm_port_on   : Marking portage_t as a permissive domain                              " -> ok
    Executing step "perm_port_off  : Removing permissive mark from portage_t                               " -> ok
    Executing step "selogin_modify : Modifying a SELinux login definition                                  " -> failed

Then, on a test system where the new package has been installed, the same testset is executed (together with all other tests) to validate if the problem is fixed.

Pushing out the new release

Finally, with the fixes in and validated, the new release is pushed out (into ~arch first of course) and the bugs are marked as RESOLVED:TEST-REQUEST. Users can confirm that it works (which would move it to VERIFIED:TEST-REQUEST) or we stabilize it after the regular testing period is over (which moves it to RESOLVED:FIXED or VERIFIED:FIXED).

I do still have to get used to Gentoo using git as its repository now. The workflow to use is documented though. Luckily, because I often get that the git push fails (due to changes to the tree since my last pull). So I need to run git pull --rebase=preserve followed by repoman full and then the push again sufficiently quick after each other).

This simple flow is easy to get used to. Thanks to the existing foundation for package maintenance (such as epatch for patching, live ebuilds that can be immediately used for releases and the ability to just cherry pick patches towards our repository) we can serve updates with just a few minutes of work.

Bernard Cafarelli a.k.a. voyageur (homepage, bugs)
LLVM 3.7.0 released, Gentoo package changes (September 02, 2015, 11:53 UTC)

As announced here, the new LLVM release is now available. If you are interested after reading the release notes (or the clang ones), I just added the corresponding packages in Gentoo, although still masked for some additional testing. mesa-11.0 release candidates compile fine with it though.
2015/09/04 edit: llvm 3.7.0 is now unmasked!

On the Gentoo side, this version bump has some nice changes:

  • We now use the cmake/ninja build system: huge thanks to fellow developer mgorny for his hard work on this rewrite. This is the recommended build system upstream now, and some features will only be available with it
  • LLDB, the debugger  is now available via USE=lldb. This is not a separate package as the build system for llvm and subprojects is quite monolithic. clang is in the same situation: sys-devel/clang package currently does almost nothing, the magic is done in llvm[clang])
  • These new ebuilds include the usual series of fixes, including some recent ones by gienah

On the TODO list, some tests are still failing on 32bit multilib (bug #558604), ocaml doc compilation/installation has a strange bug (see the additional call on docs/ocaml_doc target in the ebuild). And let’s not forget adding other tools, like polly. Another recent suggestion: add llvm’s OpenMP library needed for OpenMP with clang.

Also, if people were still using the dragonegg GCC plugin, note that it is not released anymore upstream. As 3.6.x versions had compilation problems, this package received its last rites.

Matthew Thode a.k.a. prometheanfire (homepage, bugs)
Gentoo Hardened ZFS rootfs with dm-crypt/luks 0.6.5 (September 02, 2015, 05:00 UTC)

Disclaimer

  1. Keep in mind that ZFS on Linux is supported upstream, for differing values of support
  2. I do not care much for hibernate, normal suspending works.
  3. This is for a laptop/desktop, so I choose multilib.
  4. IANAL
  5. If you patch the kernel to add in ZFS support directly, you cannot share the binary, the cddl and gpl2 are not compatible in that way.

Initialization

Make sure your installation media supports zfs on linux and installing whatever bootloader is required (uefi needs media that supports it as well). It does not matter too much if it is older media, as long as it supports what you need, Gentoo is nice like that. Here is an iso that works well for me at this link Live DVDs newer then 12.1 should also have support.

Formatting

I will be assuming the following.

  1. /boot on /dev/sda1
  2. cryptroot on /dev/sda2
  3. An optional bios_boot partition for grub (gpt stuff)
  4. swap inside cryptroot OR not used.

When using GPT for partitioning, create the first partition at 1M, just to make sure you are on a sector boundry Most newer drives are 4k advanced format drives. Because of this you need ashift=12, some/most newer SSDs need ashift=13 or greater compression set to lz4 will make your system incompatible with upstream (oracle) zfs, if you want to stay compatible then just set compression=on

General Setup

# setup encrypted partition
# aes-xts-plain64 was chosen due to speed, xts-essiv SHOULD be more secure, but about half as slow, on aes-ni I was getting about 200MBps
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/sda2
cryptsetup luksOpen /dev/sda2 cryptroot

# setup ZFS
# a deeper look into this can be found at [this link](https://github.com/ryao/zfs-overlay/blob/master/zfs-install "ryao's repo")
zpool create -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -m none -R /mnt/gentoo mypool /dev/mapper/cryptroot
zfs create -o mountpoint=none -o compression=lz4 mypool/ROOT
# rootfs
zfs create -o mountpoint=/ mypool/ROOT/rootfs
# system mountpoints were seperated so that we can set nodev and nosuid as mount options
zfs create -o mountpoint=/opt mypool/ROOT/rootfs/OPT
zfs create -o mountpoint=/usr mypool/ROOT/rootfs/USR
zfs create -o mountpoint=/usr/src -o sync=disabled mypool/ROOT/rootfs/USR/SRC
zfs create -o mountpoint=/var mypool/ROOT/rootfs/VAR
# portage
zfs create -o mountpoint=none -o setuid=off mypool/GENTOO
zfs create -o mountpoint=/usr/portage -o atime=off mypool/GENTOO/portage
zfs create -o mountpoint=/usr/portage/distfiles -o compression=off mypool/GENTOO/distfiles
zfs create -o mountpoint=/usr/portage/packages -o compression=off mypool/GENTOO/packages
zfs create -o mountpoint=/var/tmp/portage -o sync=disabled mypool/GENTOO/build-dir
# homedirs
zfs create -o mountpoint=/home mypool/HOME
zfs create -o mountpoint=/root mypool/HOME/root
# replace USER with your username
zfs create -o mountpoint=/home/USER mypool/HOME/USER
# set the bootfs, some initrams require this
zpool set bootfs=mypool/ROOT/rootfs mypool

cd /mnt/gentoo

# Download the latest stage3 and extract it.
wget ftp://gentoo.osuosl.org/pub/gentoo/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-*.tar.bz2
tar -xf /mnt/gentoo/stage3-amd64-hardened-*.tar.bz2 -C /mnt/gentoo

# get the latest portage tree
emerge --sync

# copy the zfs cache from the live system to the chroot
mkdir -p /mnt/gentoo/etc/zfs
cp /tmp/zpool.cache /mnt/gentoo/etc/zfs/zpool.cache

Kernel Config

If you are compiling the modules into the kernel staticly, then keep these things in mind.

  • When configuring the kernel, make sure that CONFIG_SPL and CONFIG_ZFS are set to 'Y'.
  • Portage will want to install sys-kernel/spl and sys-fs/zfs-kmod when emerge sys-fs/zfs is run because of dependencies. Also, both are still necessary to make the sys-fs/zfs configure script happy.
  • You do not need to run or install module-rebuild.

Install as normal up until the kernel install.

echo "=sys-kernel/genkernel-3.4.40 ~amd64       #needed for zfs and encryption support" >> /etc/portage/package.accept_keywords
emerge sys-kernel/genkernel
emerge sys-kernel/gentoo-sources                #or hardned-sources

# patch the kernel

# If you want to build the modules into the kernel directly, you will need to patch the kernel directly.  Otherwise, skip the patch commands.
# configure the kernel then prepare for zfs / spl
cd /usr/src/linux
make prepare
make scripts
env EXTRA_ECONF='--enable-linux-builtin' ebuild /usr/portage/sys-kernel/spl/spl-0.6.5.ebuild clean configure
(cd /var/tmp/portage/sys-kernel/spl-0.6.5/work/spl-0.6.5/ && ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin --with-spl-obj=/usr/src/linux' ebuild /usr/portage/sys-fs/zfs-kmod/zfs-kmod-0.6.5.ebuild clean configure
(cd /var/tmp/portage/sys-fs/zfs-kmod-0.6.5  /work/zfs-kmod-0.6.5 && ./copy-builtin /usr/src/linux)
env EXTRA_ECONF='--with-spl=/usr/src/linux --enable-linux-builtin' ebuild /usr/portage/sys-fs/zfs/zfs-0.6.5.ebuild clean merge
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' >> /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' >> /etc/portage/package.use

# finish configuring, building and installing the kernel making sure to enable dm-crypt support

# if not building zfs into the kernel, install module-rebuild
emerge module-rebuild

# install SPL and ZFS stuff zfs pulls in spl automatically
mkdir -p /etc/portage/profile
echo 'sys-fs/zfs -kernel-builtin' >> /etc/portage/profile/package.use.mask
echo 'sys-fs/zfs kernel-builtin' >> /etc/portage/package.use
emerge sys-fs/zfs

# Add zfs to the correct runlevel
rc-update add zfs-mount boot

# initrd creation, add '--callback="module-rebuild rebuild"' to the options if not building the modules into the kernel
genkernel --luks --zfs --disklabel initramfs

Finish installing as normal, your kernel line should look like this, and you should also have a the initrd defined.

# kernel line for grub2, libzfs support is not needed in grub2 because you are not mounting the filesystem directly.
linux  /kernel-3.5.0-gentoo real_root=ZFS=mypool/ROOT/rootfs crypt_root=/dev/sda2 dozfs=force ro
initrd /initramfs-genkernel-x86_64-3.5.0

In /etc/fstab, make sure BOOT, ROOT and SWAP lines are commented out and finish the install.

You should now have a working encryped zfs install.

August 28, 2015
Michael Palimaka a.k.a. kensington (homepage, bugs)

Background

For those that are not already aware, KDE’s release structure has evolved. The familiar all-in-one release of KDE SC 4 has been split into three distinct components, each with their own release cycles – KDE Frameworks 5, KDE Plasma 5, and KDE Applications 5.

This means there’s no such thing as KDE 5!

KDE Frameworks 5

KDE Frameworks 5 is a collection of libraries upon which Plasma and Applications are built. Each framework is distinct in terms of functionality, allowing consumers to depend on smaller individual libraries. This has driven adoption in other Qt-based projects such as LXQt as they no longer have to worry about “pulling in KDE”.

We ship the latest version of KDE Frameworks 5 in the main tree, and plan to target it for stabilisation shortly.

KDE Plasma 5

KDE Plasma 5 is the next generation of the Plasma desktop environment. While some might not consider it as mature as Plasma 4, it is in a good state for general use and is shipped as stable by a number of other distributions.

We ship the latest version of KDE Plasma 5 in the main tree, and would expect to target it for stabilisation within a few months.

KDE Applications 5

KDE Applications 5 consists of the remaining applications and supporting libraries. Porting is gradual process, with each new major release containing more KF5-based and fewer KDE 4-based packages.

Unfortunately, current Applications releases are not entirely coherent – some packages have features that require unreleased dependencies, and some cannot be installed at the same time as others. This situation is expected to improve in future releases as porting efforts progress.

Because of this, it’s not possible to ship KDE Applications in its entirety. Rather, we are in the process of cherry-picking known-good packages into the main tree. We have not discussed any stabilisation plan yet.

Migrating

As Frameworks are just libraries, they are automatically pulled in as required by consuming packages, and no user intervention is required.

To upgrade to Plasma 5, please follow the upgrade guide. Unfortunately it’s not possible to have both Plasma 4 and Plasma 5 installed at the same time, due to an upstream design decision.

Applications appear in the tree as a regular version bump, so will upgrade automatically.

Ongoing KDE 4 support

Plasma 4 has reached end-of-life upstream, and no further releases are expected. As per usual, we will keep it for a reasonable time before removing it completely.

As each Applications upgrade should be invisible, there’s less need to retain old versions. It is likely that the existing policy of removing old shortly after stabilisation will continue.

What is this 15.08.0 stuff, or, why is it upgrading to KDE 5?

As described above, Applications are now released separately from Plasma, following a yy.mm.xx versioning scheme. This means that, regardless of whether they are KDE 4 or KF5-based, they will work correctly in both Plasma 4 and Plasma 5, or any other desktop environment.

It is the natural upgrade path, and there is no longer a “special relationship” between Plasma and Applications the way there was in KDE SC 4.

Feedback

As always, feedback is appreciated – especially during major transitions like this. Sharing your experience will help improve it for the next person, and substantial improvements have already been made made thanks to the contributions of early testers.

Feel free to file a bug, send a mail, or drop by #gentoo-kde for a chat any time. Thanks for flying Gentoo KDE!

August 27, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
py3status v2.6 (August 27, 2015, 17:04 UTC)

Ok I was a bit too hasty in my legacy module support code clean up and I broke quite a few things on the latest version 2.5 release sorry ! :(

highlights

  • make use of pkgutil to detect properly installed modules even when they are zipped in egg files (manual install)
  • add back legacy modules output support (tuple of position / response)
  • new uname module inspired from issue 117 thanks to @ndalliard
  • remove dead code

thanks !

  • @coelebs on IRC for reporting, testing and the good spirit :)
  • @ndalliard on github for the issue, debug and for inspiring the uname module
  • @Horgix for responding to issues faster than me !

August 25, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Slowly converting from GuideXML to HTML (August 25, 2015, 09:30 UTC)

Gentoo has removed its support of the older GuideXML format in favor of using the Gentoo Wiki and a new content management system for the main site (or is it static pages, I don't have the faintest idea to be honest). I do still have a few GuideXML pages in my development space, which I am going to move to HTML pretty soon.

In order to do so, I make use of the guidexml2wiki stylesheet I developed. But instead of migrating it to wiki syntax, I want to end with HTML.

So what I do is first convert the file from GuideXML to MediaWiki with xsltproc.

Next, I use pandoc to convert this to restructured text. The idea is that the main pages on my devpage are now restructured text based. I was hoping to use markdown, but the conversion from markdown to HTML is not what I hoped it was.

The restructured text is then converted to HTML using rst2html.py. In the end, I use the following function (for conversion, once):

# Convert GuideXML to RestructedText and to HTML
gxml2html() {
  basefile=${1%%.xml};

  # Convert to Mediawiki syntax
  xsltproc ~/dev-cvs/gentoo/xml/htdocs/xsl/guidexml2wiki.xsl $1 > ${basefile}.mediawiki

  if [ -f ${basefile}.mediawiki ] ; then
    # Convert to restructured text
    pandoc -f mediawiki -t rst -s -S -o ${basefile}.rst ${basefile}.mediawiki;
  fi

  if [ -f ${basefile}.rst ] ; then
    # Use your own stylesheet links (use full https URLs for this)
    rst2html.py  --stylesheet=link-to-bootstrap.min.css,link-to-tyrian.min.css --link-stylesheet ${basefile}.rst ${basefile}.html
  fi
}

Is it perfect? No, but it works.

August 17, 2015
Alexys Jacob a.k.a. ultrabug (homepage, bugs)
py3status v2.5 (August 17, 2015, 15:05 UTC)

This new py3status comes with an amazing number of contributions and new modules !

24 files changed, 1900 insertions(+), 263 deletions(-)

I’m also glad to say that py3status becomes my first commit in the new git repository of Gentoo Linux !

IMPORTANT

Please note that this version has deprecated the legacy implicit module loading support to favour and focus on the generic i3status order += module loading/ordering !

New modules

  • new aws_bill module, by Anthony Brodard
  • new dropboxd_status module, by Tjaart van der Walt
  • new external_script module, by Dominik
  • new nvidia_temp module for displaying NVIDIA GPUs’ temperature, by J.M. Dana
  • new rate_counter module, by Amaury Brisou
  • new screenshot module, by Amaury Brisou
  • new static_string module, by Dominik
  • new taskwarrior module, by James Smith
  • new volume_status module, by Jan T.
  • new whatismyip module displaying your public/external IP as well as your online status

Changelog

As usual, full changelog is available here.

Contributors

Along with all those who reported issues and helped fixed them, quick and surely not exhaustive list:

  • Anthony Brodard
  • Tjaart van der Walt
  • Dominik
  • J.M. Dana
  • Amaury Brisou
  • James Smith
  • Jan T.
  • Zopieux
  • Horgix
  • hlmtre

What’s next ?

Well something tells me @Horgix is working hard on some standardization and on the core of py3status ! I’m sure some very interesting stuff will emerge from this, so thank you !

August 13, 2015
Sven Vermeulen a.k.a. swift (homepage, bugs)
Finding a good compression utility (August 13, 2015, 17:15 UTC)

I recently came across a wiki page written by Herman Brule which gives a quick benchmark on a couple of compression methods / algorithms. It gave me the idea of writing a quick script that tests out a wide number of compression utilities available in Gentoo (usually through the app-arch category), with also a number of options (in case multiple options are possible).

The currently supported packages are:

app-arch/bloscpack      app-arch/bzip2          app-arch/freeze
app-arch/gzip           app-arch/lha            app-arch/lrzip
app-arch/lz4            app-arch/lzip           app-arch/lzma
app-arch/lzop           app-arch/mscompress     app-arch/p7zip
app-arch/pigz           app-arch/pixz           app-arch/plzip
app-arch/pxz            app-arch/rar            app-arch/rzip
app-arch/xar            app-arch/xz-utils       app-arch/zopfli
app-arch/zpaq

The script should keep the best compression information: duration, compression ratio, compression command, as well as the compressed file itself.

Finding the "best" compression

It is not my intention to find the most optimal compression, as that would require heuristic optimizations (which has triggered my interest in seeking such software, or writing it myself) while trying out various optimization parameters.

No, what I want is to find the "best" compression for a given file, with "best" being either

  • most reduced size (which I call compression delta in my script)
  • best reduction obtained per time unit (which I call the efficiency)

For me personally, I think I would use it for the various raw image files that I have through the photography hobby. Those image files are difficult to compress (the Nikon DS3200 I use is an entry-level camera which applies lossy compression already for its raw files) but their total size is considerable, and it would allow me to better use the storage I have available both on my laptop (which is SSD-only) as well as backup server.

But next to the best compression ratio, the efficiency is also an important metric as it shows how efficient the algorithm works in a certain time aspect. If one compression method yields 80% reduction in 5 minutes, and another one yields 80,5% in 45 minutes, then I might want to prefer the first one even though that is not the best compression at all.

Although the script could be used to get the most compression (without resolving to an optimization algorithm for the compression commands) for each file, this is definitely not the use case. A single run can take hours for files that are compressed in a handful of seconds. But it can show the best algorithms for a particular file type (for instance, do a few runs on a couple of raw image files and see which method is most succesful).

Another use case I'm currently looking into is how much improvement I can get when multiple files (all raw image files) are first grouped in a single archive (.tar). Theoretically, this should improve the compression, but by how much?

How the script works

The script does not contain much intelligence. It iterates over a wide set of compression commands that I tested out, checks the final compressed file size, and if it is better than a previous one it keeps this compressed file (and its statistics).

I tried to group some of the compressions together based on the algorithm used, but as I don't really know the details of the algorithms (it's based on manual pages and internet sites) and some of them combine multiple algorithms, it is more of a high-level selection than anything else.

The script can also only run the compressions of a single application (which I use when I'm fine-tuning the parameter runs).

A run shows something like the following:

Original file (test.nef) size 20958430 bytes
      package name                                                 command      duration                   size compr.Δ effic.:
      ------------                                                 -------      --------                   ---- ------- -------
app-arch/bloscpack                                               blpk -n 4           0.1               20947097 0.00054 0.00416
app-arch/bloscpack                                               blpk -n 8           0.1               20947097 0.00054 0.00492
app-arch/bloscpack                                              blpk -n 16           0.1               20947097 0.00054 0.00492
    app-arch/bzip2                                                   bzip2           2.0               19285616 0.07982 0.03991
    app-arch/bzip2                                                bzip2 -1           2.0               19881886 0.05137 0.02543
    app-arch/bzip2                                                bzip2 -2           1.9               19673083 0.06133 0.03211
...
    app-arch/p7zip                                      7za -tzip -mm=PPMd           5.9               19002882 0.09331 0.01592
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=24           5.7               19002882 0.09331 0.01640
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=25           6.4               18871933 0.09955 0.01551
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=26           7.7               18771632 0.10434 0.01364
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=27           9.0               18652402 0.11003 0.01224
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=28          10.0               18521291 0.11628 0.01161
    app-arch/p7zip                                       7za -t7z -m0=PPMd           5.7               18999088 0.09349 0.01634
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=24           5.8               18999088 0.09349 0.01617
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=25           6.5               18868478 0.09972 0.01534
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=26           7.5               18770031 0.10442 0.01387
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=27           8.6               18651294 0.11008 0.01282
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=28          10.6               18518330 0.11643 0.01100
      app-arch/rar                                                     rar           0.9               20249470 0.03383 0.03980
      app-arch/rar                                                 rar -m0           0.0               20958497 -0.00000        -0.00008
      app-arch/rar                                                 rar -m1           0.2               20243598 0.03411 0.14829
      app-arch/rar                                                 rar -m2           0.8               20252266 0.03369 0.04433
      app-arch/rar                                                 rar -m3           0.8               20249470 0.03383 0.04027
      app-arch/rar                                                 rar -m4           0.9               20248859 0.03386 0.03983
      app-arch/rar                                                 rar -m5           0.8               20248577 0.03387 0.04181
    app-arch/lrzip                                                lrzip -z          13.1               19769417 0.05673 0.00432
     app-arch/zpaq                                                    zpaq           0.2               20970029 -0.00055        -0.00252
The best compression was found with 7za -t7z -m0=PPMd:mem=28.
The compression delta obtained was 0.11643 within 10.58 seconds.
This file is now available as test.nef.7z.

In the above example, the test file was around 20 MByte. The best compression compression command that the script found was:

~$ 7za -t7z -m0=PPMd:mem=28 a test.nef.7z test.nef

The resulting file (test.nef.7z) is 18 MByte, a reduction of 11,64%. The compression command took almost 11 seconds to do its thing, which gave an efficiency rating of 0,011, which is definitely not a fast one.

Some other algorithms don't do bad either with a better efficiency. For instance:

   app-arch/pbzip2                                                  pbzip2           0.6               19287402 0.07973 0.13071

In this case, the pbzip2 command got almost 8% reduction in less than a second, which is considerably more efficient than the 11-seconds long 7za run.

Want to try it out yourself?

I've pushed the script to my github location. Do a quick review of the code first (to see that I did not include anything malicious) and then execute it to see how it works:

~$ sw_comprbest -h
Usage: sw_comprbest --infile=<inputfile> [--family=<family>[,...]] [--command=<cmd>]
       sw_comprbest -i <inputfile> [-f <family>[,...]] [-c <cmd>]

Supported families: blosc bwt deflate lzma ppmd zpaq. These can be provided comma-separated.
Command is an additional filter - only the tests that use this base command are run.

The output shows
  - The package (in Gentoo) that the command belongs to
  - The command run
  - The duration (in seconds)
  - The size (in bytes) of the resulting file
  - The compression delta (percentage) showing how much is reduced (higher is better)
  - The efficiency ratio showing how much reduction (percentage) per second (higher is better)

When the command supports multithreading, we use the number of available cores on the system (as told by /proc/cpuinfo).

For instance, to try it out against a PDF file:

~$ sw_comprbest -i MEA6-Sven_Vermeulen-Research_Summary.pdf
Original file (MEA6-Sven_Vermeulen-Research_Summary.pdf) size 117763 bytes
...
The best compression was found with zopfli --deflate.
The compression delta obtained was 0.00982 within 0.19 seconds.
This file is now available as MEA6-Sven_Vermeulen-Research_Summary.pdf.deflate.

So in this case, the resulting file is hardly better compressed - the PDF itself is already compressed. Let's try it against the uncompressed PDF:

~$ pdftk MEA6-Sven_Vermeulen-Research_Summary.pdf output test.pdf uncompress
~$ sw_comprbest -i test.pdf
Original file (test.pdf) size 144670 bytes
...
The best compression was found with lrzip -z.
The compression delta obtained was 0.27739 within 0.18 seconds.
This file is now available as test.pdf.lrz.

This is somewhat better:

~$ ls -l MEA6-Sven_Vermeulen-Research_Summary.pdf* test.pdf*
-rw-r--r--. 1 swift swift 117763 Aug  7 14:32 MEA6-Sven_Vermeulen-Research_Summary.pdf
-rw-r--r--. 1 swift swift 116606 Aug  7 14:32 MEA6-Sven_Vermeulen-Research_Summary.pdf.deflate
-rw-r--r--. 1 swift swift 144670 Aug  7 14:34 test.pdf
-rw-r--r--. 1 swift swift 104540 Aug  7 14:35 test.pdf.lrz

The resulting file is 11,22% reduced from the original one.

August 12, 2015
Gentoo Package Repository now using Git (August 12, 2015, 00:00 UTC)

Good things come to those who wait: The main Gentoo package repository (also known as the Portage tree or by its historic name gentoo-x86) is now based on Git.

Timeline

The Gentoo Git migration has arrived and is expected to be completed soon. As previously announced, the CVS freeze occurred on 8 August and Git commits for developers were opened soon after. As a last step, rsync mirrors are expected to have the updated changelogs again on or after 12 August. Read-only access to gentoo-x86 (and write to the other CVS repositories) was restored on 9 August following the freeze.

History

Work on migrating the repository from CVS to Git began in 2006 with a proof-of-concept migration project during the Summer of Code. Back then, migrating the repository took a week and using Git was considerably slower than using CVS. While plans to move were shelved for a while, things improved over the coming years. Several features were implemented in Git, Portage, and other tools to meet the requirements of a migrated repository.

What changes?

The repository can be checked out from git.gentoo.org and is available via our Git web interface.

For users of our package repository, nothing changes: Updates continue to be available via the established mechanisms (rsync, webrsync, snapshots). Options to fetch the package tree via Git are to be announced later.

The migration facilitates the process of new contributors getting involved as proxy maintainers and eventually Developers. Alternate places for users to submit pull requests, such as GitHub, can be expected in the future.

In regards to package signing, the migration will streamline how GPG keys are used. This will allow end-to-end signature tracking from the developer to the final repository, as outlined in GLEP 57 et seq.

While the last issues are being worked on, join us in thanking everyone involved in the project. As always, you can discuss this on our Forums or hit up @Gentoo.

August 07, 2015
Nathan Zachary a.k.a. nathanzachary (homepage, bugs)

Important!

My tech articles—especially Linux ones—are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Recently, I’ve spent a huge amount of time working on Apache and PHP-FPM in order to allow for a threaded Apache MPM whilst still using PHP and virtual hosts (vhosts). As this article is going to be rather lengthy, I’m going to split it up into sections below. It’s my hope that after reading the article, you’ll be able to take advantage of Apache’s newer Event MPM and mod_proxy_fcgi in order to get the best performance out of your web server.

1. Definitions:

Before delving into the overarching problem, it might be best to have some definitions in place. If you’re new to the whole idea of web servers, programming languages used for web sites, and such, these definitions may help you become more acquainted with the problem that this article addresses. If you’re familiar with things like Apache, PHP, threading, preforking, and so on, feel free to skip this basic section.

  • Web server – on a physical server, the web server is an application that actually hands out (or serves) web pages to clients as they request them from their browser. Apache and nginx are two popular web servers in the Linux / UNIX world.
  • Apache MPM – a multi-processing module (or MPM) is the pluggable mechanism by which Apache binds to network ports on a server, processes incoming requests for web sites, and creates children to handle each request.
    • Prefork – the older Apache MPM that isn’t threaded, and deals with each request by creating a completely separate Apache process. It is needed for programming languages and libraries that don’t deal well with threading (like some PHP modules).
    • Event – a newer Apache MPM that is threaded, which allows Apache to handle more requests at a time by passing off some of the work to threads, so that the master process can work on other things.
  • Virtual host (vhost) – a method of deploying multiple websites on the same physical server via the same web server application. Basically, you could have site1.com and site2.com both on the same server, and even using the same IP address.
  • PHP – PHP is arguably one of the most common programming languages used for website creation. Many web content management systems (like WordPress, Joomla!, and MediaWiki) are written in PHP.
  • mod_php – An Apache module for processing PHP. With this module, the PHP interpreter is essentially embedded within the Apache application. It is typically used with the Prefork MPM.
  • mod_proxy_fcgi – An Apache module that allows for passing off operations to a FastCGI processor (PHP can be interpreted this way via PHP-FPM).
  • PHP-FPM – Stands for PHP FastCGI Process Manager, and is exactly what it sounds like: a manager for PHP that’s being interpreted via FastCGI. Unlike mod_php, this means that PHP is being interpreted by FastCGI, and not directly within the Apache application.
  • UNIX socket – a mechanism that allows two processes within the same UNIX/Linux operating system to communicate with one another.

Defining threads versus processes in detail is beyond the scope of this article, but basically, a thread is much smaller and less resource-intense than a process. There are many benefits to using threads instead of full processes for smaller tasks.

2. Introduction:

Okay, so now that we have a few operational definitions in place, let’s get to the underlying problem that this article addresses. When running one or more busy websites on a server, it’s desirable to reduce the amount of processing power and memory that is needed to serve those sites. Doing so will, consequently, allow the sites to be served more efficiently, rapidly, and to more simultaneous clients. One great way to do that is to switch from the Prefork MPM to the new Event MPM in Apache 2.4. However, that requires getting rid of mod_php and switching to a PHP-FPM backend proxied via mod_proxy_fcgi. All that sounds fine and dandy, except that there have been several problems with it in the past (such as .htaccess files not being honoured, effectively passing the processed PHP file back to Apache, and making the whole system work with virtual hosts [vhosts]). Later, I will show you a method that addresses these problems, but beforehand, it might help to see some diagrams that outline these two different MPMs and their respective connections to PHP process.

The de facto way to use PHP and Apache has been with the embedded PHP processor via mod_php:

Apache Prefork MPM with mod_php embedded
Apache with the Prefork MPM and embedded mod_php
Click to enlarge

 

This new method involves proxying the PHP processing to PHP-FPM:

Apache Event MPM to PHP-FPM via mod_proxy_fcgi
Apache with the Event MPM offloading to PHP-FPM via mod_proxy_fcgi
Click to enlarge

3. The Setup:

On the majority of my servers, I use Gentoo Linux, and as such, the exact file locations or methods for each task may be different on your server(s). Despite the distro-specific differences, though, the overall configuration should be mostly the same.

3a. OS and Linux kernel:

Before jumping into the package rebuilds needed to swap Apache to the Event MPM and to use PHP-FPM, a few OS/kernel-related items should be confirmed or modified: 1) epoll support, 2) the maximum number of open file descriptors, and 3) the number of backlogged sockets.

Epoll support
The Apache Event MPM and PHP-FPM can use several different event mechanisms, but on a Linux system, it is advisable to use epoll. Even though epoll has been the default since the 2.6 branch of the Linux kernel, you should still confirm that it is available on your system. That can be done with two commands (one for kernel support, and one for glibc support):

Kernel support for epoll (replace /usr/src/linux/.config with the actual location of your kernel config):
# grep -i epoll /usr/src/linux/.config
CONFIG_EPOLL=y

glibc support for epoll
# nm -D /lib/libc.so.6 | grep -i epoll
00000000000e7dc0 T epoll_create
00000000000e7df0 T epoll_create1
00000000000e7e20 T epoll_ctl
00000000000e7a40 T epoll_pwait
00000000000e7e50 T epoll_wait

Max open file descriptors
Another aspect you will need to tune based on your specific needs is the maximum number of open file descriptors. Basically, since there will be files opened for the Apache connections, the UNIX socket to PHP-FPM, and for the PHP-FPM processes themselves, it is a good idea to have a high maximum number of open files. You can check the current kernel-imposed limit by using the following command:

# cat /proc/sys/fs/file-max
3291373

Though that number is usually very high, you also need to check the limits based on user. For sake of ease, and seeing as servers running many vhosts may frequently add and modify user accounts, I generally set the max file descriptors pretty high for *all* users.

# grep nofile /etc/security/limits.conf
# - nofile - max number of open file descriptors
* soft nofile 16384
* hard nofile 16384

The specifics of modifying the limits.conf file are outside of the scope of this article, but the above changes will set the soft and hard limits to 16,384 open files for any user on the system.

Backlogged socket connections
The last OS configuration that you might want to initially configure is the maximum number of backlogged socket connections. This number is important because we will be proxying Apache to PHP-FPM via UNIX sockets, and this value sets the maximum number of connections per socket. It should be noted that this is the maximum number per socket, so setting this kernel parameter to some outlandish value doesn’t really make a lot of sense. As you will see later in this tutorial, I create a socket for each virtual host, so unless you plan on having more than 1024 simultaneous connections to PHP-FPM per site, this value is sufficient. You can change it permanently in /etc/sysctl.conf:

# grep somaxconn /etc/sysctl.conf
net.core.somaxconn = 1024

and remember to run sysctl -p thereafter in order to make the changes active.

3b. Package rebuilds:

Now that you’ve checked (and possibly modified) some OS-related configurations, a few packages needed to be recompiled in order to change from the Apache Prefork MPM to the Event MPM and to use PHP-FPM. Here’s a list of the changes that I needed to make:

  • In /etc/portage/make.conf
    • Change the MPM to read APACHE2_MPMS="event"
    • Ensure you have at least the following APACHE2_MODULES enabled: cgid proxy proxy_fcgi
    • Per the mod_cgid documentation you should enable mod_cgid and disable mod_cgi when using the Event MPM
  • In /etc/portage/package.use
    • Apache: www-servers/apache threads
    • PHP must include at least: dev-lang/php cgi fpm threads
      • Note that you do not need to have the ‘apache2’ USE flag enabled for PHP since you’re not using Apache’s PHP module
      • However, you will still need to have the MIME types and the DirectoryIndex directives set (see subsection 3d below)
    • Eselect for PHP must have: app-eselect/eselect-php fpm

Now that those changes have been made, and you’ve recompiled the appropriate packages (Apache, and PHP), it’s time to start configuring the packages before reloading them. In Gentoo, these changes won’t take effect until you restart the running instances. For me, that meant that I could compile everything ahead of time, and then make the configuration changes listed in the remaining 3.x subsections before restarting. That procedure allowed for very little downtime!

3c. Apache:

Within the configurations for the main Apache application, there are several changes that need to be made due to switching to a threaded version with the Event MPM. The first three bullet points for httpd.conf listed below relate to the proxy modules, and the fourth bullet point relates to the directives specific to the Event MPM. For more information about the MPM directives, see Apache’s documentation.

  • In /etc/apache2/httpd.conf
    • The proxy and proxy_fcgi modules need to be loaded:
      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
    • Comment out LoadModule cgi_module modules/mod_cgi.so so it does not get loaded
    • Replace that with the one for mod_cgid: LoadModule cgid_module modules/mod_cgid.so
    • Update the MPM portion for Event:
      • <IfModule event.c>
        StartServers 2
        ServerLimit 16
        MinSpareThreads 75
        MaxSpareThreads 250
        ThreadsPerChild 25
        MaxRequestWorkers 400
        MaxConnectionsPerChild 10000
        </IfModule>

You will also need to make sure that Apache loads certain modules at runtime. In Gentoo, that configuration is done in /etc/conf.d/apache, and needs to contain at least the two modules listed below (note that your configuration will likely have other modules loaded as well):

  • In /etc/conf.d/apache
    • APACHE2_OPTS="-D PHP5 -D PROXY"

3d. Apache PHP module:

The last change that needs to be made for Apache is to the configuration file for how it handles PHP processing (in Gentoo, this configuration is found in /etc/apache2/modules.d/70_mod_php5.conf. I would suggest making a backup of the existing file, and making changes to a copy so that it can be easily reverted, if necessary.

70_mod_php5.conf BEFORE changes
<IfDefine PHP5>
  <IfModule !mod_php5.c>
    LoadModule php5_module modules/libphp5.so
  </IfModule>
  <FilesMatch "\.(php|php5|phtml)$">
    SetHandler application/x-httpd-php
  </FilesMatch>
  <FilesMatch "\.phps$">
    SetHandler application/x-httpd-php-source
  </FilesMatch>

  DirectoryIndex index.php index.phtml
</IfDefine>

70_mod_php5.conf AFTER changes
<IfDefine PHP5>
  ## Define FilesMatch in each vhost
  <IfModule mod_mime.c>
    AddHandler application/x-httpd-php .php .php5 .phtml
    AddHandler application/x-httpd-php-source .phps
  </IfModule>

  DirectoryIndex index.php index.phtml
</IfDefine>

Basically, you’re getting rid of the LoadModule reference for mod_php, and all the FileMatch directives. Now, if you’re not using virtual hosts, you can put the FilesMatch and Proxy directives from the vhosts section (3e) below in your main PHP module configuration. That being said, the point of this article is to set up everything for use with vhosts, and as such, the module should look like the example immediately above.

As mentioned in section 3b above, the reason you still need the 70_mod_php5.conf at all is so that you can definite the MIME types for PHP and the DirectoryIndex. If you would rather not have this file present, (since you’re not really using mod_php), you can just place these directives (the AddHandler and DirectoryIndex ones) in your main Apache configuration—your choice.

3e. vhosts:

For each vhost, I would recommend creating a user and corresponding group. How you manage local users and groups on your server is beyond the scope of this document. For this example, though, we’re going to have site1.com and site2.com (very creative, I know 😎 ). To keep it simple, the corresponding users will be ‘site1’ and ‘site2’, respectively. For each site, you will have a separate vhost configuration (and again, how you manage your vhosts is beyond the scope of this document). Within each vhost, the big change that you need to make is to add the FilesMatch and Proxy directives. Here’s the generic template for the vhost configuration additions:

Generic template for vhost configs
<FilesMatch "\.php$">
  SetHandler "proxy:unix:///var/run/php-fpm/$pool.sock|fcgi://$pool/"
</FilesMatch>

<Proxy fcgi://$pool/ enablereuse=on max=10>
</Proxy>

In that template, you will replace $pool with the name of each PHP-FPM pool, which will be explained in the next subsection (3f). In my opinion, it is easiest to keep the name of the pool the same as the user assigned to each site, but you’re free to change those naming conventions so that they make sense to you. Based on my site1.com / site2.com example, the vhost configuration additions would be:

site1.com vhost config
<FilesMatch "\.php$">
  SetHandler "proxy:unix:///var/run/php-fpm/site1.sock|fcgi://site1/"
</FilesMatch>

<Proxy fcgi://site1/ enablereuse=on max=10>
</Proxy>

site2.com vhost config
<FilesMatch "\.php$">
  SetHandler "proxy:unix:///var/run/php-fpm/site2.sock|fcgi://site2/"
</FilesMatch>

<Proxy fcgi://site2/ enablereuse=on max=10>
</Proxy>

3f. PHP-FPM:

Since you re-compiled PHP with FPM support (in step 3b), you need to actually start the new process with /etc/init.d/php-fpm start and add it to the default runlevel so that it starts automatically when the server boots (note that this command varies based on your init system):

rc-update add php-fpm default (Gentoo with OpenRC)

Now it’s time to actually configure PHP-FPM via the php-fpm.conf file, which, in Gentoo, is located at /etc/php/fpm-php$VERSION/php-fpm.conf. There are two sections to this configuration file: 1) global directives, which apply to all PHP-FPM instances, and 2) pool directives, which can be changed for each pool (e.g. each vhost). For various non-mandatory options, please see the full list of PHP-FPM directives.

At the top of php-fpm.conf are the global directives, which are:
[global]
error_log = /var/log/php-fpm.log
events.mechanism = epoll
emergency_restart_threshold = 0

These directives should be fairly self-explanatory, but here’s a quick summary:

  • [global] – The intro block stating that the following directives are global and not pool-specific
  • error_log – The file to which PHP-FPM will log errors and other notices
  • events.mechanism – What should PHP-FPM use to process events (relates to the epoll portion in subsection 3a)
  • emergency_restart_threshold – How many PHP-FPM children must die improperly before it automatically restarts (0 means the threshold is disabled)

Thereafter are the pool directives, for which I follow this template, with one pool for each vhost:
;; $SITE.$TLD
[$USER]
listen = /var/run/php-fpm/$USER.sock
listen.owner = $USER
listen.group = apache
listen.mode = 0660
user = $USER
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

It may look a bit intimidating at first, but only the $SITE, $TLD, and $USER variables change based on your vhosts and corresponding users. When the template is used for the site1.com / site2.com example, the pools look like:

;; site1.com
[site1]
listen = /var/run/php-fpm/site1.sock
listen.owner = site1
listen.group = apache
listen.mode = 0660
user = site1
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

;; site2.com
[site2]
listen = /var/run/php-fpm/site2.sock
listen.owner = site2
listen.group = apache
listen.mode = 0660
user = site2
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

Since the UNIX sockets are created in /var/run/php-fpm/, you need to make sure that that directory exists and is owned by root. The listen.owner and listen.group directives will make the actual UNIX socket for the pool owned by $USER:apache, and will make sure that the apache group can write to it (which is necessary).

The following directives are ones that you will want to change based on site traffic and server resources:

  • pm.start_servers – The number of PHP-FPM children that should be spawned automatically
  • pm.max_children – The maximum number of children allowed (connection limit)
  • pm.min_spare_servers – The minimum number of spare idle PHP-FPM servers to have available
  • pm.max_spare_servers – The maximum number of spare idle PHP-FPM servers to have available
  • pm.max_requests – Maximum number of requests each child should handle before re-spawning
  • pm.request_terminate_timeout – Maximum amount of time to process a request (similar to max_execution_time in php.ini

These directives should all be adjusted based on the needs of each site. For instance, a really busy site with many, many, simultaneous PHP connections may need 3 servers each with 100 children, whilst a low-traffic site may only only need 1 server with 10 children. The thing to remember is that those children are only active whilst actually processing PHP, which could be a very short amount of time (possibly measured in milliseconds). Contrapositively, setting the numbers too high for the server to handle (in terms of available memory and processor) will result in poor performance when sites are under load. As with anything, tuning the pool requirements will take time and analysis to get it right.

Remember to reload (or restart) PHP-FPM after making any changes to its configuration (global or pool-based):

/etc/init.d/php-fpm reload

4. Verification:

After you have configured Apache, your vhosts, PHP, PHP-FPM, and the other components mentioned throughout this article, you will need to restart them all (just for sake of cleanness). To verify that things are as they should be, you can simply browse to one of the sites configured in your vhosts and make sure that it functions as intended. You can also verify some of the individual components with the following commands:

Apache supports threads, and is using the Event MPM
# apache2 -V | grep 'Server MPM\|threaded'
Server MPM:     event
  threaded:     yes (fixed thread count)

Apache / vhost syntax
# apache2ctl configtest
 * Checking apache2 configuration ...     [ ok ]

5. Troubleshooting:

So you followed the directions here, and it didn’t go flawlessly?! Hark! 😮 In all seriousness, there are several potential points of failure for setting up Apache to communicate with PHP-FPM via mod_proxy_fcgi, but fortunately, there are some methods for troubleshooting (one of the beauties of Linux and other UNIX derivatives is logging).

By default, mod_proxy and mod_proxy_fcgi will report errors to the log that you have specified via the ErrorLog directive, either in your overall Apache configuration, or within each vhost. Those logs serve as a good starting point for tracking down problems. For instance, before I had appropriately set the UNIX socket permission options in php-fpm.conf, I noticed these errors:

[Wed Jul 08 00:03:26.717538 2015] [proxy:error] [pid 2582:tid 140159555143424] (13)Permission denied: AH02454: FCGI: attempt to connect to Unix domain socket /var/run/php-fpm/site1.sock (*) failed
[Wed Jul 08 00:03:26.717548 2015] [proxy_fcgi:error] [pid 2582:tid 140159555143424] [client 52350] AH01079: failed to make connection to backend: httpd-UDS

That error in bold text indicated that Apache didn’t have permission to write to the UNIX socket. Setting the permissions accordingly—see the listen.owner and listen.group directives portion of subsection 3f—fixed the problem.

Say that the errors don’t provide detailed enough information, though. You can set the LogLevel directive (again, either in your overall Apache configuration, or per vhost) to substantially increase the verbosity—to debug or even trace levels. You can even change the log level just for certain modules, so that you don’t get bogged down with information that is irrelevant to your particular error. For instance:

LogLevel warn proxy:debug proxy_fcgi:debug

will set the overall LogLevel to “warn,” but the LogLevel for mod_proxy and mod_proxy_fcgi to “debug,” in order to make them more verbose.

6. Conclusion:

If you’ve stuck with me throughout this gigantic post, you’re either really interested in systems engineering and the web stack, or you may have some borderline masochistic tendencies. 😛 I hope that you have found the article helpful in setting up PHP to proxy PHP interpretation to PHP-FPM via mod_proxy_fcgi, and that you’re able to see the benefits of this type of offloading. Not only does this method of processing PHP files free up resources so that Apache can serve more simultaneous connections with fewer resources, but it also more closely adheres to the UNIX philosophy of having applications perform only one task, and do so in the best way possible.

If you have any questions, comments, concerns, or suggestions, please feel free to leave a comment.

Cheers,
Zach