Planet Gentoo

Jorge Manuel B. S. Vicetto: Amarok-2.2.0/1 and >=binutils-2.20

Jorge Manuel B. S. Vicetto (jmbsvicetto) — Fri, 20 Nov 2009 03:28:59 +0000

I've just bumped media-sound/amarok in the tree to 2.2.1-r1 which includes some missing deps.
Even though 2.2.1 hit the tree before the official release, I'm a few days behind in the bump and unmasking it - the mask should be out when you read this.
Unfortunately, amarok-2.2.0 and amarok-2.2.1 segfault on start when built with >=binutils-2.20 even though they have no issue starting with >=binutils-2.20 if built with a previous version of binutils - see bug #290662 for more details.
While the issue isn't solved, here is a quick workaround:

# emerge -1 =binutils-2.19.1-r1 && emerge -1 amarok && emerge -1 binutils

Original post blogged on b2evolution.

Christian Faulhammer: Kernel testing request 2.6.31

Christian Faulhammer (fauli) — Thu, 19 Nov 2009 09:19:35 +0000

Another testing request. Kernel 2.6.31 is planned to be stabled soon on all architectures. Hereby I now ask users on stable x86 systems to upgrade to gentoo-sources 2.6.31-r6 or vanilla-sources 2.6.31.6 and report back to me in all cases (failure or success). The other arches appreciate tests, too, but I can't speak for them. Thanks in advance.

Alex Alexander: Qt 4.6.0 rc1 – in portage – binary incompatibility warning!

Alex Alexander (wired) — Thu, 19 Nov 2009 08:06:23 +0000

So… qt-*-4.6.0_rc1 is now in the portage tree, masked (since its not a final release) and you can begin testing your shiny Qt applications with it.

But there’s a catch.

4.6.0_rc1 is not binary compatible with 4.6.0_beta1. [1]
This means that if you’re upgrading from 4.6.0_beta1 you have to rebuild every single app depending on qt-*, or they won’t start at all. This obviously includes all of KDE 4.

You can get portage to do that by using a command like this:
*update* it seems some shells didn’t like the old command, so I’ve updated it:

emerge -av1 $(for i in $(qlist -IC x11-libs/qt-); do equery -q d $i | grep -v 'x11-libs/qt-' | sed "s/^/=/"; done)

Note that this command assumes your system is up-to-date. If some installed packages don’t have ebuilds available for their version anymore, you’ll have to resolve that manually (probably by removing them and rerunning the command).

You’ll find equery in app-portage/gentoolkit and qlist in app-portage/portage-utils.

You don’t need to do this if you’re upgrading from Qt 4.5.3 (although you should). You’ll also be safe if you upgrade from 4.5.3 to 4.6.0 final when the time comes.

Portage will warn you about this when you upgrade qt-core to 4.6.0_rc1:

* Messages for package x11-libs/qt-core-4.6.0_rc1:

*
* Binary compatibility broke between 4.6.0_beta1 and 4.6.0_rc1.
* If you are upgrading from 4.6.0_beta1, you’ll have to
* re-emerge everything that depends on Qt.
* Use the following command:
*
* emerge -av1 $(for pkg in $(equery -q d \
* $(qlist -IC x11-libs/qt-) | grep -v “x11-libs/qt-” |
* sort -u); do echo “=$pkg”; done)
*
* YOU’VE BEEN WARNED
*

But users (my user side as well :p) tend to ignore ewarns from time to time

Happy rebuilding!

[1] http://labs.trolltech.com/blogs/2009/11/12/bc-break-in-46-against-previous-46/

Patrick Lauer: CPU Hotswapping and how to disable processors

Patrick Lauer (bonsaikitten) — Wed, 18 Nov 2009 23:43:58 +0000

Here's something awesome I found mostly by accident:
In recent kernels the support for hotswapping CPUs works on x86/amd64 architectures. I stumbled over it in the 2.6.32 menuconfig and couldn't wonder if it actually works. So I had a look and found this gem:

# cat /proc/interrupts | grep CPU
            CPU0       CPU1       CPU2       CPU3

Very boring, 4 processors.

echo 0 > /sys/devices/system/cpu/cpu3/online

And we just knocked out one!
We see that in dmesg:

kvm: disabling virtualization on CPU3                                                                                                                       
CPU 3 is now offline

Hmm, are you thinking what I'm thinking?

kvm: disabling virtualization on CPU2                                                                                                                       
CPU 2 is now offline                                                                                                                                        
kvm: disabling virtualization on CPU1                                                                                                                       
CPU 1 is now offline                                                                                                                                        
SMP alternatives: switching to UP code

Wheeee. I just castrated it to a single core! I actually didn't check if the kernel lets me take CPU0 offline. That would be hilarious. Anyway ...

echo  > /sys/devices/system/cpu/cpu1/online

And we just gained a CPU:

SMP alternatives: switching to SMP code                                                                                                                     
Booting processor 1 APIC 0x1 ip 0x6000                                                                                                                      
Initializing CPU#1                                                                                                                                          
Calibrating delay using timer specific routine.. 5200.20 BogoMIPS (lpj=10400418)                                                                            
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)                                                                                           
CPU: L2 Cache: 512K (64 bytes/line)                                                                                                                         
CPU 1/0x1 -> Node 0                                                                                                                                         
CPU: Physical Processor ID: 0                                                                                                                               
CPU: Processor Core ID: 1                                                                                                                                   
CPU1: AMD Phenom(tm) 9950 Quad-Core Processor stepping 03                                                                                                   
checking TSC synchronization [CPU#0 -> CPU#1]: passed.                                                                                                      
kvm: enabling virtualization on CPU1

This is seriously wicked. Now I just need to figure out how to bolt that onto powermanagement so that the machine knocks out cores when idle and powersaves. Linux never gets boring ...

Diego E. Pettenò: What distributions want

Diego E. Pettenò (flameeyes) — Wed, 18 Nov 2009 12:58:20 +0000

Or A 101 lesson on how to ensure that your software package is available to the distribution users (which, incidentally, are the Linux users; while already the conglomerate Linux marketshare is pretty low when compared with Windows and OS X, the marketshare of not-really-distributions like Slackware or Linux from scratch is probably so trivial that you don’t have to care about them most of the time. That, and their users are usually not so much scared about installing stuff by themselves.

I’m posting this quickie because I’d like to tell one other thing to Ryan… yes, you are a drama queen. And you proven it right with your latest rant and it really upsets me that instead of trying to understand the problem your solution seems to be closing yourslef even more inside your little world. It upsets me because I can see you as a capable programmer and I’d prefer your capacity being used for something that people can benefit from, rather than wasted on stuff pointless, like FatELF.

The fact that instead of trying to understand the technical points that me and others made, and tell us why you think they are not good enough, you’re just closing yourself further. By saying that “lots of people talked about it” you’re just proving what you’re looking for: fame and glory. Without actual substantial results to back it up. Just an hint: the people who matters aren’t those who continue saying “FatELF will make distributions useless, will make it possible to develop cross-platform software, will solve the world’s hunger”; the people who matters are those that review FatELF for its technical side, and most of us already deemed it pointless; I already explained what I think about it.

Any ISV that thinks FatELF will solve their cross-distribution or cross-architecture problems have no idea what an ELF file is; they don’t really understand the whole situation at all. I’m pretty sure there are such ISV out there… but I wouldn’t really look forward for them to decide what to put inside the kernel and the other projects.

Do you want your software (your games) to be available to as many people as possible? Start working with the freaking distributions! You don’t need to have mastered all the possible package managers, you don’t even need to know about any of them directly, but you got to listen if packagers ask for some changes. If a packager asks you to unbundle a library or allow selecting between bundled or system library; do it, they have their reasons and they know how to deal with eventual incompatibilities. If a packager asks you to either change your installation structure or at least make it flexible, that’s because with a very few exceptions, distributions are fine with following the FHS.

Take a look to “Distributions-friendly packages”: part 1 part 2 and part 3 .

But no, Ryan’s solution here is again taking cheap shots to distributions and packagers, without actually noticing that, after more than ten years distributions are not going away .

Oh and the first commenter who will try to say again that FatELF is the solution, can please tell me how’s that going to ensure that the people writing the code will understand the difference between little endian and big endian? Or that the size of a pointer is not always 32-bit? Count that in as a captcha; if you cannot give me an answer to those two questions, your comment supporting FatELF as The Solution will be deleted.

Sune Kloppenborg Jeppesen: Watching Freeview HD DVB-T MPEG-4 channels

Sune Kloppenborg Jeppesen (jaervosz) — Wed, 18 Nov 2009 07:29:54 +0000

Last week before catching the H1N1 I had the chance to play around a bit with DVB-T.

Here in .dk the analog TV signal was turned off at the start of the month, and we got some new digital freeview channels. Most notably DR HD broadcasting unencrypted 720p movies in MPEG-4.

The MPEG-4 encoding can be a bit tricky since TVs from just a few years back (ie. both my Panasonic TVs) only support MPEG-2 decoding, so a lot of people had to buy new hardware MPEG-4 decoders. However not wanting yet another power consuming device I decided to solve it on the cheap (not speaking of yet another redundant remote control).

First off old DVB-T PCI cards/USB sticks should generally be able to receive DVB-T broadcasts in any encoding, including MPEG-4. I just had an old Hauppauge Nova-T DVB-T card lying around in the basement, what luck:)

I've collected a few notes here that might be helpful for others.

First figure out what kernel options you'll have to enable.

Getting the right channel list:
emerge linuxtv-dvb-apps linuxtv-dvb-firmware w_scan w_scan -x > dvb_frequencies dvbscan dvb_frequencies > channels.conf

Now watch the first channel with: vlc channels.conf. Note if vlc doesn't play anything you might have messed up the generation of channels.conf as I somehow managed to do the first time.

Then record the HD content:
gnutv -channels channels.conf -out file drhd.mp4 "DR HD"

Now come the not so great part, it seems like the MPEG-4 used by DR HD is not well supported in Linux or more precisely the HE-AAC audio codec.

Current stable vlc-1.0.2 seems to be slow to tune in to the programs and audio is choppy and stastistics show dropped audio frames. Upgrading to vlc-1.0.3 seems to make it tune in faster, but the audio is still choppy.

Changing the "Caching value in ms" to 1000 seems to have solved most of the stuttering problems (Preferences -> All -> Input/Codec -> Access modules -> DVB).

Current stable ffmpeg-0.5-r1 only shows 3 streams and refuses to encode:
Stream #0.0[0x907]: Video: h264, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 50 tbr, 90k tbn, 100 tbc Stream #0.1[0x91f](dan): Subtitle: dvbsub Stream #0.2[0x920](dan): Subtitle: dvbsub

However upgrading ffmpeg to 0.5_p20373, results in it detecting another 2 streams:
Stream #0.0[0x7ef]: Data: 0x0006 Stream #0.1[0x907]: Video: h264, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 50 tbr, 90k tbn, 100 tbc Stream #0.2[0x911](dan): Data: 0x0011 Stream #0.3[0x91f](dan): Subtitle: dvbsub Stream #0.4[0x920](dan): Subtitle: dvbsub

And the live ffmpeg ebuild actually detects the audio stream, but hangs when encoding:
Stream #0.0[0x7ef](dan): Subtitle: 0x0006 Stream #0.1[0x907]: Video: h264, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 50 tbr, 90k tbn, 100 tbc Stream #0.2[0x911](dan): Audio: aac, 44100 Hz, 2 channels, s16, 450 kb/s Stream #0.3[0x91f](dan): Subtitle: dvbsub Stream #0.4[0x920](dan): Subtitle: dvbsub

So for now I can watch DR HD on one computer, but playback on my PCH-110 is not working either as the audio codec HE-AAC seems to be unsupported too. Later on I might just install a MythTV backend and do some automatic transcoding to the troublesome devices. That is once proper HE-AAC support is in ffmpeg:-)

Original post blogged on b2evolution.

Patrick Lauer: "Random crashes" with glibc 2.10 and 2.11

Patrick Lauer (bonsaikitten) — Tue, 17 Nov 2009 19:15:18 +0000

As documented in this bug (which mirrors an upstream bug here there's a bug in glibc 2.10 and 2.11, and this one seems to be easy to hit. Multithreaded apps "randomly" crash with "Invalid free" and other confusing errors. A hackaround is to unset or empty the environment variable "MALLOC_CHECK_". For me setting MALLOC_CHECK_="" before starting some of the affected packages seems to completely hide the error, now we can only hope that the gentoo glibc gets this patch soon.

Patrick Lauer: Awesome portage options

Patrick Lauer (bonsaikitten) — Tue, 17 Nov 2009 18:47:10 +0000

For the rest of this post I'll only consider portage 2.2. Most options are in portage 2.1 already, but I'm a lazy bum, so I don't compare to see what's what.

You can set PORTAGE_DEFAULT_OPS in /etc/make.conf, but if you add --ask you will have trouble running emerge from a script. --ignore-default-opts disables those defaults so you can run emerge --sync in a cronjob again.

Sets are really great, --list-sets shows you which are available. Just have a look, there are some nice ones - "security", "installed", "unavailable" ... they can help streamline some tasks. I find their names quite self-explanatory.

If you want to put something into the world file without rebuilding it use --noreplace, and if you want to remove it again use --deselect.

--nospinner disables that funny rotating spinner thingy so you can save precious bandwidth when connected remotely, and --quiet hides most of the output, which can be nice if you don't want to be hypnotized by scrolling compile output. For the OCD crowd --quiet-build might be nice as it doesn't show the compile output on console, but redirects to logfiles. --changelog is neat for seeing the log messages for that update, this often shows fixed bugs or other issues you might care about. --color with a parameter y or n toggles colorized output. And of course --alphabetical. The horror of unsorted output!

Sometimes people are confused that emerge -e world tries to update packages that emerge -uND world misses. That is usually caused by build-only dependencies. --with-bdeps=y and --complete-graph are good options to modify portage behaviour.

If you're on a fast machine and in a hurry you can try to set --jobs X with a reasonable value of X. Think about memory needs and such before setting it to infinity minus one! With --keep-going it gets really easy to not have the whole process stopped on the first failed package. This is not without issues, but it avoids the --resume --skipfirst in a loop tricks. If --jobs seems to hard to calibrate to you --load-average=LOAD may help to limit it.

For the scripters --columns might be nice, it tweaks the output to be more script friendly.

Support for binary packages has grown considerably, there's support for local (-k / -K) and remote (-g / -G ) binpkg repositories. And you can --buildpkg and --buildpkgonly to create them (they are stored in PKGDIR). There's --binpkg-respect-use to only install the packages that have useflags set the same as the current configuration - it's a very powerful mechanism if you need to support Gentoo on multiple machines and don't want to compile that much.

I hope y'all enjoyed this little lesson in RTFM, there's plenty of other options to discover. Don't be afraid of the documentation, it doesn't bite and makes your life easier :)

Patrick Lauer: Configuring Portage

Patrick Lauer (bonsaikitten) — Mon, 16 Nov 2009 15:29:32 +0000

Few people take the time to actually read through the documentation, but if you have some time to spare "man make.conf" is a great read.
For example you can pre-set some CLI options like --ask or --verbose in EMERGE_DEFAULT_OPTS so you never have to type them again. Especially the FEATURES variable has some interesting bits:

buildpkg builds packages of everything
buildsyspkg builds only packages of the system set, which is awesome for recovery and doesn't take much space.

keepwork keeps the $WORKDIR and can be quite useful for debugging purposes (but not for general use)
noclean leaves even more there.
fail-clean is the opposite, it always wipes the build directories. Useful if you build on a small (but fast) disk or tmpfs.
installsources installs all the package sources to /usr/src/debug/, which can be used for debugging, but eats lots of space. Together with splitdebug it offers some really great debugging convenience.
test-fail-continue helps when you just want to have the tests run for logging purposes, but don't want the package to not be installed if tests fail. Most people won't need this.

split-elog and split-log features are quite interesting if you do logging.

Logging can be very nice to have, and portage has lots of configuration options for it.
PORTAGE_ELOG_SYSTEM defines how the log data is sent, be it through syslog, email or just to a file. Or completely custom?
And you can do combinations like PORTAGE_ELOG_SYSTEM="mail:warn,error syslog:* save".
PORTAGE_ELOG_CLASSES defines what you want to log - warnings, errors, qa warnings, everything ... it's your choice.

Of course there are lots of other configuration options:
PORTAGE_NICENESS can be useful when you don't want portage to interfere with anything else.
PORTAGE_IONICE_COMMAND needs ionice (or an equivalent tool) and can be used to make the disk activity of portage a bit less distracting. Both features may increase the time needed to install things, but will make portage more benign so you can still do things while it runs.

Also you can change almost all directories - PORTDIR, DISTDIR, PKGDIR and so on. This allows you to make portage behave a lot more like you want it (unless the defaults satisfy you already ...)

Diego E. Pettenò: Virtualisation WTF once again.

Diego E. Pettenò (flameeyes) — Mon, 16 Nov 2009 10:36:27 +0000

To test some more RTSP clients I’ve been working to get more virtual machines available in my system; to do so I first extended the space available in my system by connecting one more half-a-terabyte hard drive (removing the DVD burner from Yamato), and then started again working on a proper init script for KVM/Qemu (as Pavel already asked me before, and provided me with an example).

Speaking about it, if somebody were to send my way an USB or FireWire DVD burner I’d be probably quite happy; while I have other three DVD burners around – iMac, MacBook Pro and Compaq laptop – having one on Yamato from time to time came out useful; not necessary, so wasting a SATA port for it was not really a good idea after all, but still useful.

I started writing a simple script before leaving for my vacation and extended it a bit more yesterday. But in line with the usual virtualisation woes the results aren’t excessively positive:

FreeBSD 8 pre-releases no longer seem to kernel panic when run in qemu (the last beta I tried did, the latest rc available does not); on the other hand it does seem to have problems with the default network (it works if started after boot but not at boot); it works fine with e1000;
NetBSD still is a desperate case: with qemu (and VDE) no network seem to work; e1000 is not even recognised, while the others end up timing out, silently or not; this is without ACPI enabled, if I do enable ACPI, no network card seems to be detected; with KVM, it freezes, no matter with or without ACPI, during boot up;
Pavel already suggested a method using socat and the monitor socket for qemu to shut down the VM cleanly; the shutdown request will cause the qemu or kvm instance to send the ACPI signal (if configured!) and then it would shut down cleanly… the problem is that the method requires socat, which is quite broken (even in the 2-beta branch).

Let me explain what the problem is with socat: its build system tries to identify the size of various POD types that are used by the code; to do so it uses some autoconf trickery, the -Werror switch and relies on pointer comparison to work with two POD types of the same size, even if different. Guess what? That’s no longer the case. A warning sign was already present: the code started failing some time ago when -Wall was added to the flags, so the ebuild strips it. Does that tell you something?

I looked into sanitizing the test; the proper solution would be to use run-test, rather than build-tests, for what I can see; but even if that’s possible, it’s quite intrusive and it breaks cross-compilation. So I went to look why the thing really needed to find the equivalents… and the result is that the code is definitely messy. It’s designed to work on pre-standard systems, and keep compatible with so many different operating systems that fixing the build system up is going to require quite a bit of code hacking as well.

It would be much easier if netcat supported handling of unix local sockets, but no implementation I have used seem to. My solution to this problem is to replace socat with something else; based on a scripting language, such as Perl so that’s as portable, and at the same time less prone to problems like those socat is facing now. I asked a few people to see if they can write up a replacement, hopefully this will bring us a decent replacement so we can kill that.

So if you’re interested in having a vm init script that works with Gentoo without having to deal with stuff like libvirt and so on, then you should probably find a way to coordinate all together and get a socat replacement done.

Patrick Lauer: HOWTO Radeon (opensource) + R700 + 3D / OpenGL

Patrick Lauer (bonsaikitten) — Sun, 15 Nov 2009 18:08:48 +0000

Finally I stopped slacking for long enough to fix a few bits of my desktop, and the results are grrrrrrreat.
Now I have (OpenGL!) full effects in KDE4 as opposed to the slightly less bouncy XRender-accelerated thingies before. Performance is pretty awesome (but then the HD4650 shouldn't even notice those few effects).
What you need:

~arch install (I'm not going to care to find out what minimal versions you need)
x11 overlay
a really recent kernel

And with really recent I mean "at least 2.6.32". At the time of writing that hasn't been released, so a 2.6.32-rc6 git-sources has to substitute for me. From what I've read you might have to disable framebuffer for things to work well, but as I'm usually seeing text mode for ~30 seconds every month I don't care enough to find out. I'm lazy!
In the kernel config you need to enable DRM and especially the radeon bits. Device Drivers -> Graphics -> Direct Rendering Manager is the "most important" bit there.
The following packages were suggested in a few places, I have no idea if that is the minimal set. But you'll have to unmask:

>=x11-libs/libdrm-9999
>=media-libs/mesa-9999
>=x11-base/xorg-server-9999
>=x11-proto/fixesproto-9999
>=x11-proto/xextproto-9999
>=x11-proto/xf86vidmodeproto-9999
>=x11-proto/renderproto-9999
>=x11-proto/recordproto-9999
>=x11-proto/inputproto-9999
>=x11-proto/xineramaproto-9999
>=x11-proto/bigreqsproto-9999
>=x11-proto/xf86driproto-9999
>=x11-proto/xf86dgaproto-9999
>=x11-proto/xcmiscproto-9999
>=x11-base/xorg-drivers-9999
>=x11-libs/libXext-9999
>=x11-libs/libXi-9999
>=x11-proto/xproto-9999
>=x11-libs/libX11-9999
>=x11-libs/libxcb-9999
>=x11-proto/xcb-proto-9999

Now go forth and rebuild all your shiny new packages.
If you managed to build that and reboot your new kernel things should look pretty much as before. The only "obvious" hints I've found to test are the output of glxinfo (has changed quite a bit) and that KDE4 allows me to use OpenGL now. And maybe the wobbly windows effect was a giveaway :)

I'm positively surprised that things have progressed this far, and I'm happy to finally be able to use more of my graphics card :)

EDIT: Seems that this is not the minimal set of packages and configuration needed. Some people suggested -9999 packages of mesa + libdrm + xf86-video-ati only. If that works even better :)

Gilles Dartiguelongue: Looking for a padawan

Gilles Dartiguelongue (eva) — Fri, 13 Nov 2009 23:58:33 +0000

http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=1&chap=2

Title says it all. It's pretty obvious that gnome team can't currently keep up with the income of new bugs (especially a lot of unexpected 2.26 bugs) and I keep being split between my gnome duties and maintenance of various other stuff (notably freedesktop that no-one seems to care about these days). So I need one, two or more guys/gals that want to get their hands a bit dirty. If you feel up to it, just come and say hello on #gentoo-desktop, if I'm not around, probably other gnome herd members will be waiting for you .

Original post blogged on b2evolution.

Doug Goldstein: NVIDIA legacy drivers update

Doug Goldstein (cardoe) — Fri, 13 Nov 2009 20:57:42 +0000

As I previously noted, the 96.x.y and 173.x.y series had not seen any updates yet for xorg-server 1.7 compatibility, however today that has changed. They both saw updates late last night from NVIDIA and now those drivers are available in the Gentoo tree so feel free to give them a try.

Patrick Lauer: The OOM Killer (and how to make it less annoying)

Patrick Lauer (bonsaikitten) — Fri, 13 Nov 2009 12:20:59 +0000

The linux kernel has lots of complexity in memory management. Swap allows to go beyond the size of real memory to allow applications to use "more". But still, at some point, you might exhaust all available memory.

The next application that requests memory (usually through malloc) will cause the kernel some trouble. It can either deny the request (which often causes hilarious results in the application) or free some memory somehow. (About the hilarity: Many coders assume that a malloc will always succeed. If it doesn't you'll get interesting misbehaviour like segmentation faults. Lots of fun to debug ...)

So, how does the kernel free memory? It can't just ask some other processes to surrender some. But it can terminate processes! It's a terminally stupid idea, but it's so stupid that it often works. And the handler for that is, obviously, the out-of-memory killer.
There's a very nice bit of information hidden in /proc to tell you what the oom-killer would do if it had to run now.

/proc/$pid/oom_score

contains the current value of the process with PID $pid. You could just compare them and see who is good and who is bad. And you can adjust it - a rarely used protection, but it might just help the oom-killer to act more sanely and less psychotic.

/proc/$pid/oom_adj

That's a numerical value used as a multiplier. Valid values are in the range -16 to +15, plus the special value -17, which disables oom-killing altogether for this process. The heuristic is quite complex, to quote:

The process to be killed in an out-of-memory situation is selected among all others
based on its badness score. This value equals the original memory size of the process
and is then updated according to its CPU time (utime + stime) and the
run time (uptime - start time). The longer it runs the smaller is the score.
Badness score is divided by the square root of the CPU time and then by
the double square root of the run time.
Swapped out tasks are killed first. Half of each child's memory size is added to
the parent's score if they do not share the same memory. Thus forking servers
are the prime candidates to be killed. Having only one 'hungry' child will make
parent less preferable than the child.

On some systems you might not ever want to have the oom-killer strike. It's just a hilariously bad idea to kill random processes. And you can even disable it:

The sysctl vm.overcommit_memory variable (also represented in /proc/sys/vm/overcommit_memory ) defines the behaviour. To summarize: 0 is default, where the kernel uses some heuristics and allows allocating more memory than available (which is what can trigger the nice OOM assassin) 1 always allows overcommit. The documentation is a bit sparse, but it seems to be tuned by vm.overcommit_ratio, which gives a percentage to overcommit (unless I misread that). And finally a value of 2 disables overcommitting and limits application memory to the size of (swap + ram*ratio). This means that worst case you'll disallow a request when there's still physical memory available, but you'll never have to trigger Mr.OOM-Killer.

What is best? That depends on what you do and how you want things to fail. overcommit_memory = 2 will cause memory allocation failures, but your machine will always be "alive". overcommit_memory = 0 might allow to allocate more memory, but you risk getting any process killed by oom. Sucks to have sshd killed on a server - maybe it's not the best idea to have a psychotic process assassin? But it's your choice, so do what you want to do :)

Josh Saddler: November Xfce desktop

Josh Saddler (nightmorph) — Thu, 12 Nov 2009 19:23:23 +0000

Decided I'd shake things up a bit this month, after keeping the same look for nearly three straight months. Thus, I present:

icons: Area o.43
gtk+: Rele (Rezlooks engine)
xfwm4: Rezlooks-gtk (yes, it is confusingly named)
background: rassilon

It's grungy, but rather sleek. Surprisingly easy on the eyes, too. The lighter elements of the Rele gtk+ theme aren't overpoweringly white, but are just light enough to provide a decent contrast to the generally darker Area icons.

There's also an uncluttered version here.

I've been looking to assemble themes that are grungy, and themes that are warmer-yet-wintry. I like winter. I still have some hope that these 80 degree weeks will come to an end soon. We're almost to the middle of November. Surely we'll see gray sky, cool breezes, and maybe even rain here in SoCal at some point, right? Right? Well, if not, I can at least put it on my desktop.

Original post from Planet Gentoo.

Alex Alexander: how-to: keep your kde 3.5 after it’s removed from gentoo’s tree, using the kde-sunset overlay

Alex Alexander (wired) — Wed, 11 Nov 2009 23:31:51 +0000

As most of you know, KDE 3.5 is getting off the Gentoo train for various reasons mentioned elsewhere (in short: security, lack of upstream interest).

Some people are still using it though and they have been asking how they can keep it on their system, without emerge screaming each time they try to update world.

I’m writing this post to guide those people (you? ) on how they can achieve that.

1. Adding the kde-sunset overlay
Perhaps the most important thing you need to remember is that the ebuilds will be completely removed from the tree, so you need to add an overlay called kde-sunset to portage which contains a copy of most of them.

To add the overlay you need to have layman with git USE flag enabled:

USE="git" emerge -av layman

make sure you add the git USE flag in your package.use or make.conf

you can then use layman to install the overlay:

layman -f layman -a kde-sunset

if this is your first time using layman, you’ll have to add the following line @ the end of your make.conf:

source /usr/local/portage/layman/make.conf

you can keep the overlay updated by running:

layman -s kde-sunset

2. KDE 3.5 is masked (this is necessary only while KDE 3.5 is still in tree)
Now we have the overlay, but until KDE 3.5 is completely wiped off the tree, there’s a mask to warn users about it.
Portage will scream about this mask each time you try to emerge so you need to unmask KDE 3.5 manually.

There are two (and a half) ways to do this:
a1) if your package.unmask is a folder, symlink the unmask file from kde-sunset to /etc/portage/package.unmask/

ln -s /usr/local/portage/layman/kde-sunset/Documentation/package.unmask/kde-3.5 /etc/portage/package.unmask/kde-3.5

a2) if your package.unmask is not a folder you can append the file (but you’ll have to update it manually)

cat /usr/local/portage/layman/kde-sunset/Documentation/package.unmask/kde-3.5 >> /etc/portage/package.unmask

b) if you don’t like all that, you can emerge autounmask

emerge -av autounmask

and ask it to unmask stuff for you – note that autounmask doesn’t like :slot and will only work with -version

autounmask kde-meta-3.5

3. I don’t want KDE 4 (yet)
So all is great now but portage wants to install KDE 4 when you emerge world.
This is happening because KDE 4 went stable recently and portage thinks you want to upgrade (you should btw ).
To fix this one we need to tell portage that we actually want the 3.5 version of KDE and we don’t care about 4.

we need to edit /var/lib/portage/world and add a version (or a slot) to every kde 3.5 app we have in there.

* be careful with this file *

for example, lets say that you installed kde 3.5 using the kde-meta ebuild.
if you open /var/lib/portage/world you’ll find a line saying

kde-base/kde-meta

you need to add :3.5 which is the 3.5 slot, or -3.5.10 which is the version (or both, version first) at the end. if you add the version, you’ll have to prepend a “=” as well.

the line should now read:

kde-base/kde-meta:3.5

do this for each KDE 3.5 app you have in the world file. If you’re unsure of the slot/version, use eix packagename to find it out.

4. All systems go

emerge -avDuN world

^^ this should now be clean of any KDE 4 traces and mask warnings

If you have any issues or are interested in maintaining kde-sunset, you may visit us over @ #gentoo-kde in freenode’s IRC network or drop us an email at kde _at_ gentoo _dot_ org. We’ll do our best to help you

Maybe after reading this post your KDE 3 installation will last a bit longer

Enjoy and see you w/ KDE 4 soon

Josh Saddler: Tabu Audio Player

Josh Saddler (nightmorph) — Wed, 11 Nov 2009 14:50:05 +0000

Even though I'm currently sick with H1N1, also known as swine flu, this morning I was feeling well enough to write an ebuild for an interesting media app I found: Tabu Audio Player. It's an interesting player -- while it still needs some translation work into English, it's simple and has an appealing UI drawn by Cairo.

It had only a single configure option, "debug," and a very short list of dependencies, so I figured it'd be a simple ebuild to write, right?

Wrong! Turns out that after a half-hours' worth of poking at configure.ac, the package's check for --disable-debug/--enable-debug was completely broken. If you explicitly passed --disable-debug, like "-debug" in IUSE, then it would enable the debug build every time. Thanks to rej and a3li on IRC for nailing this problem down; they were really helpful.

a3li also shared his solution for the same problem in one of his packages, so I used it in my ebuild to ensure that "debug" works properly as a USE variable.

Meanwhile, I sent a bug report to upstream asking for a smarter debug check. I also asked for CFLAGS="-02" to be removed from Makefile.am, that way folks are free to use their own -O levels without having to resort to using sed on the Makefile, as I had to in the ebuild.

Speaking of which: I have an ebuild for Tabu 2.1 waiting for you in my devspace, if you'd like to try it out.

Original post from Planet Gentoo.

Patrick Lauer: The mistery of swappiness

Patrick Lauer (bonsaikitten) — Tue, 10 Nov 2009 19:18:46 +0000

For the longest time operating systems have been able to handle swap. In short swap extends physical memory with slow diskspace so that applications can use more memory than there is available.
On most unix systems the swap is in a dedicated partition because that has the lowest overhead. Plus you don't risk running out of diskspace when you want to swap, so things are quite predictable and nice. Linux has a very nice knob you can turn to affect the swap policy. It will not avoid swapping (in some situations you will have to), but it will affect how and when swap is used. That knob is /proc/sys/vm/swappiness.

The kernel default is a value of 60. The value can be between 0 and 100 and is effectively a percentage. It is used roughly in the following way:
If all available memory is exhausted (application memory, buffers and filesystem cache) and any memory allocation is requested the kernel needs to free a few pages of memory. It can either swap out application memory or drop some filesystem cache. The "swappiness" knob affects the probability which one is chosen.
This means that at a swappiness of 0 the kernel will try to never swap out a process, and at 100 it will try to always swap out processes and keep the filesystem cache intact. So with the default, if you use more than ca. 40% of your memory for applications and the rest is used as filesystem cache it will already start swapping a bit. The hilarious result is that you may up swapping a lot with lots of memory left - think of a machine with 64GB RAM! If you try to use 32G memory you'll be in swap hell.
That default might have been good with machines with less than 256MB RAM, but with current desktops and servers it is usually not optimal.
Now you might be tempted to tune it down to 0. Avoid swap. Swap is slow. All is good?
Not quite. At 0 your machine will try to avoid swapping until the last moment. Then it will have killed all filesystem cache (so every file operation will hit the disks) and in addition to that you start swapping like a madman. The result is usually a "swap storm" that hits very sudden. At the point where you might need some performance your machine doesn't provide it and might just be unresponsive to your input for a few minutes.
The other end (a value near 100) might make sense for a file server, but then it might be cheaper to just not run extra services on a machine that is very loaded already. I don't really see a usecase for a swappiness of 100 except maybe on machines that are very memory-limited.
On my desktop I've found a swappiness of 10-20 to be the sweet spot. This means that when 80%+ of memory is used by applications the machine will start swapping, but it's a more gradual hit and not an instant kill. And because there's still some filesystem cache the responsiveness for starting new processes (like a login shell ;) ) is still high enough to allow recovery from this pessimal system state.
Still your goal for optimal performance should be to avoid swapping. Disk access is slower than RAM by a factor of 1000 or more!
I've seen servers achieve roughly double the throughput with the right swappiness value - it can avoid an expensive hardware upgrade. Of course that's not all the tuning advice I have, so if you wish to discuss that feel free to send me a mail and maybe I can prove to you that Gentoo is the fastest penguin out there ...

Maybe I should discuss the OOM killer too - most people have seen it, but few know who it is and why he goes killing processes.

Alex Alexander: gentoo-user-el mailing-list – for all the greeks out there :)

Alex Alexander (wired) — Tue, 10 Nov 2009 14:17:01 +0000

Starting today, Greek Gentoo users have their own mailing list to discuss stuff and ask questions: gentoo-user-el.

For more info on how to subscribe, please visit http://www.gentoo.org/main/en/lists.xml

The list is not mentioned there yet, but you can use it normally.

[the rest of this message is in greek]

Θέλετε να συζητήσετε για το αγαπημένο σας Gentoo στα ελληνικά αλλά δεν βρίσκετε κάποια official mailing list?

Πλέον μπορείτε να χρησιμοποιείτε την νέα λίστα gentoo-user-el, που δημιουργήθηκε για να καλύψει αυτό ακριβώς το κενό.

Για να γραφτείτε, αρκεί να στείλετε ένα κενό e-mail στην διεύθυνση gentoo-user-el+subscribe _at_ lists.gentoo.org και να ακολουθήσετε τις οδηγίες.

Μπορείτε να συνομιλήσετε με άλλους χρήστες του Gentoo αλλά και με Gentoo Developers (είμαστε 6 Έλληνες) και να λύσετε κάθε πρόβλημα και απορία σας

Για περισσότερες πληροφορίες σχετικά με τις λίστες του Gentoo μπορείτε να επισκευτείτε την σελίδα http://www.gentoo.org/main/en/lists.xml

Η λίστα δεν αναφέρεται ακόμα σε αυτήν την σελίδα (πρέπει να ενημερωθεί από το κατάλληλο team), αλλά εσείς μπορείτε να γραφτείτε και να την χρησιμοποιήσετε κανονικά.

Επίσης μπορείτε να διαβάσετε το περιεχόμενο της λίστας πηγαίνοντας στην διεύθυνση http://archives.gentoo.org/gentoo-user-el/

Όπως βλέπετε εγώ έκανα την αρχή: http://archives.gentoo.org/gentoo-user-el/msg_4eb9364bded3280c9685356b9e0445d0.xml

Εάν είστε από αυτούς που προτιμούν πιο άμεση επικοινωνία, μπορείτε να επισκευτείτε το κανάλι #gentoo-el στο freenode irc network: http://java.freenode.net/index.php?channel=gentoo-el

Τα λέμε εκεί

Diego E. Pettenò: Vacation time for me!

Diego E. Pettenò (flameeyes) — Tue, 10 Nov 2009 07:47:12 +0000

Just wanted everybody to note that I have a flight to catch for London today at lunch time; will be away till Sunday and will come back for all kind of work on Monday, so if you write me in the mean time I won’t be around, most likely.

Doug Goldstein: MythTV 0.22 is out

Doug Goldstein (cardoe) — Mon, 09 Nov 2009 05:37:29 +0000

As many may have already known, MythTV 0.22 is officially out. Slashdot is late to the party to report the news, however Gentoo does have ~arch ebuilds available in the tree for MythTV and all officially supported plugins. In the future I may add an ebuild as well for mythstream since it is a fairly popular plugin.

Some known issues include compilation problems on x86 as a result of PIC support, the issue is being looked into actively. For this upgrade, you must manually upgrade your database however the process is pretty painless.

If you run into any other issues, please file a bug and I’ll do my best to get to it.

Diego E. Pettenò: Plugins aren't always a good choice

Diego E. Pettenò (flameeyes) — Sun, 08 Nov 2009 16:30:34 +0000

I’ve been saying this for quite a while, probably one of the most on-topic post has been written a few months ago but there are some indications about it in posts about xine and other again.

I used to be an enthusiast about plugin interfaces; with time, though, I started having more and more doubts about their actual usefulness — it’s a tract I really much like in myself, I’m fine with reconsidering my own positions over time, deciding that I was wrong; it happened before with other things, like KDE (and C++ in general).

It’s not like I’m totally against the use of plugins altogether. I only think that they are expensive in more ways than one, and that their usefulness is often overstated, or tied to other kind of artificial limitations. For instance, dividing a software’s features over multiple plugins makes it easier for the binary distributions to package them, usually: they only have to ship a package with the main body of the software, and many for the plugins (one per plugin might actually be too much so sometimes they might be grouped). This works out pretty well for both the distribution and, usually, the user: the plugins that are not installed will not bring in extra dependencies, they won’t take time to load and they won’t use memory for either code nor data. It basically allows binary distribution to have a flexibility to compare with Gentoo’s USE flags (and similar options in almost any other source-based distribution).

But as I said this comes with costs, that might or might not be worth it in general. For instance, Luca wanted to implement plugins for feng similarly to what Apache and lighttpd have. I can understand his point: let’s not load code for the stuff we don’t have to deal with, which is more or less the same reason why Apache and lighttpd have modules; in the case of feng, if you don’t care about access log, why should you be loading the access load support at all? I can give you a couple of reasons:

because the complexity of managing a plugin to deal with the access log (or any other similar task) is higher than just having a piece of static code that handles that;
because the overhead of having a plugin loaded just to do that is higher than that of having the static code built in and not enabled into configuration.

The first problem is a result of the way a plugin interface is built: the main body of the software cannot know about its plugins in too specific ways. If the interface is a very generic plugin interface, you add some “hook locations” and then it’s the plugin’s task to find how to do its magic, not the software’s. There are some exceptions to this rule: if you have a plugin interface for handling protocols, like the KIO interface (and I think gvfs has the same) you get the protocol from the URL and call the correct plugin, but even then you’re leaving it to the plugin to deal with doing its magic. You can provide a way for the plugin to tell the main body what it needs and what it can do (like which functions it implements) but even that requires the plugins to be quite autonomous. And that means also being able to take care of allocating and freeing the resources as needed.

The second problem is not only tied to the cost of calling the dynamic linker dynamically to load the plugin and its eventual dependencies (which is a non-trivial amount of work, one has to say), also by the need for having code that deals with finding the modules to load, the loading of those modules, their initialisation, keeping a list of modules to call at any given interface point, and two more points: the PIC problem and the problem of less-than-page-sized segments. This last problem is often ignored, but it’s my main reason to dislike plugins when they are not warranted for other reasons. Given a page size of 4KiB (which is the norm on Linux for what I know), if the code is smaller than that size, it’ll still require a full page (it won’t pack with the rest of the software’s code areas); but at least code is disk-backed (if it’s PIC, of course), it’s worse for what concerns variable data, or variable relocated data, since those are not disk-backed, and it’s not rare that you’d be using a whole page for something like 100 bytes of actual variables.

In the case of the access log module that Luca wrote for feng, the statistics are as such:

flame@yamato feng % size modules/.libs/mod_accesslog.so
   text	   data	    bss	    dec	    hex	filename
   4792	    704	     16	   5512	   1588	modules/.libs/mod_accesslog.so

Which results in two pages (8KiB) for bss and data segments, neither disk-backed, and two disk-backed pages for the executable code (text): 16KiB of addressable memory for a mapping that does not reach 6KiB, it’s a 10KiB overhead, which is much higher than 50%. And that’s the memory overhead alone. The whole overhead, as you might guess at this point, is usually within 12KiB (since you got three segments, and each can have at most one byte less than page size as overhead — it’s actually more complex than this but let’s assume this is true).

It really doesn’t sound like a huge overhead by itself, but you have to always judge it compared to the size of the plugin itself. In the case of feng’s access log, you got a very young plugin that lacks a lot of functionality, so one might say that with the time it’ll be worth it… so I’d like to show you the size statistics for the Apache modules on the very server my blog is hosted. Before doing so, though, I have to remind you one huge difference: feng is built with most optimisations turned off, while Apache is built optimised for size; they are both AMD64 though so the comparison is quite easy.

flame@vanguard ~ $ size /usr/lib64/apache2/modules/*.so | sort -n -k 4
   text	   data	    bss	    dec	    hex	filename
   2529	    792	     16	   3337	    d09	/usr/lib64/apache2/modules/mod_authn_default.so
   2960	    808	     16	   3784	    ec8	/usr/lib64/apache2/modules/mod_authz_user.so
   3499	    856	     16	   4371	   1113	/usr/lib64/apache2/modules/mod_authn_file.so
   3617	    912	     16	   4545	   11c1	/usr/lib64/apache2/modules/mod_env.so
   3773	    808	     24	   4605	   11fd	/usr/lib64/apache2/modules/mod_logio.so
   4035	    888	     16	   4939	   134b	/usr/lib64/apache2/modules/mod_dir.so
   4161	    752	     80	   4993	   1381	/usr/lib64/apache2/modules/mod_unique_id.so
   4136	    888	     16	   5040	   13b0	/usr/lib64/apache2/modules/mod_actions.so
   5129	    952	     24	   6105	   17d9	/usr/lib64/apache2/modules/mod_authz_host.so
   6589	   1056	     16	   7661	   1ded	/usr/lib64/apache2/modules/mod_file_cache.so
   6826	   1024	     16	   7866	   1eba	/usr/lib64/apache2/modules/mod_expires.so
   7367	   1040	     16	   8423	   20e7	/usr/lib64/apache2/modules/mod_setenvif.so
   7519	   1064	     16	   8599	   2197	/usr/lib64/apache2/modules/mod_speling.so
   8583	   1240	     16	   9839	   266f	/usr/lib64/apache2/modules/mod_alias.so
  11006	   1168	     16	  12190	   2f9e	/usr/lib64/apache2/modules/mod_filter.so
  12269	   1184	     32	  13485	   34ad	/usr/lib64/apache2/modules/mod_headers.so
  12521	   1672	     24	  14217	   3789	/usr/lib64/apache2/modules/mod_mime.so
  15935	   1312	     16	  17263	   436f	/usr/lib64/apache2/modules/mod_deflate.so
  18150	   1392	    224	  19766	   4d36	/usr/lib64/apache2/modules/mod_log_config.so
  18358	   2040	     16	  20414	   4fbe	/usr/lib64/apache2/modules/mod_mime_magic.so
  18996	   1544	     48	  20588	   506c	/usr/lib64/apache2/modules/mod_cgi.so
  20406	   1592	     32	  22030	   560e	/usr/lib64/apache2/modules/mod_mem_cache.so
  22593	   1504	    152	  24249	   5eb9	/usr/lib64/apache2/modules/mod_auth_digest.so
  26494	   1376	     16	  27886	   6cee	/usr/lib64/apache2/modules/mod_negotiation.so
  27576	   1800	     64	  29440	   7300	/usr/lib64/apache2/modules/mod_cache.so
  54299	   2096	     80	  56475	   dc9b	/usr/lib64/apache2/modules/mod_rewrite.so
 268867	  13152	     80	 282099	  44df3	/usr/lib64/apache2/modules/mod_security2.so
 288868	  11520	    280	 300668	  4967c	/usr/lib64/apache2/modules/mod_passenger.so

The list is ordered for size of the whole plugin (summed up, not counting padding); the last three positions are definitely unsurprisingly, although it surprises me the sheer size of the two that are not part of Apache itself (and I start to wonder whether they link something in statically that I missed). The fact that the rewrite module was likely the most complex plugin in Apache’s distribution never left me.

As you can see, almost all plugins have vast overhead especially for what concerns the bss segment (all of them have at least 16 bytes used, and that warrants a whole page for them: 4080 bytes wasted each); the data segment is also interesting: only the two external ones have more than a page worth of variables (which also is suspicious to me). When all the plugins are loaded (like they most likely are right now as well on my server) there are at least 100KiB of overhead; just for the sheer fact that these are plugins and thus have their own address space. Might not sound like a lot of overhead indeed, since Apache is requesting so much memory already, especially with Passenger running, but it definitely doesn’t sound like a good thing for embedded systems.

Now I have no doubt that a lot of people like the fact that Apache has all of those as plugins as they can then use the same Apache build across different configurations without risking to have in memory more code and data than it’s actually needed, but is that right? While it’s obvious that it would be impossible to drop the plugin interface from Apache (since it’s used by third-party developers, more on that later), I would be glad if it was possible to build in the modules that come with Apache (given I can already choose which ones to build or not in Gentoo). Of course I also am using Apache with two configurations, and for instance the other one does not use the authentication system for anything, and this one is not using CGI, but is the overhead caused by the rest of modules worth the hassle, given that Apache already has a way to not initialise the unused built-ins?

I named above “third party developers” but I have to say now that it wasn’t really a proper definition, since it’s not just what third parties would do, it might very well be the original developers who might want to make use of plugins to develop separate projects for some (complex) features, and have different release handling altogether. For uses like that, the cost of plugins is often justifiable; and I am definitely not against having a plugin interface in feng. My main beef is when the plugins are created for functions that are part of the basic featureset of a software.

Another unfortunately not uncommon problem with plugins is that the interface might be skewed by bad design, like the case was (and is) for xine: when trying to open a file, it has to pass through all the plugins, so it loads all of them into memory, together with the libraries they depend on, to ask each of them to test the current file; since plugins cannot really be properly unloaded (and it’s not just a xine limitation) the memory will still be used, the libraries will still be mapped into memory (and relocated, causing copy on write, and thus, more memory) and at least half the point of using plugins has gone away (the ability to only load the code that is actually going to be used). Of course you’re left with the chance that an ABI break does not kill the whole program, but just the plugin, but that’s a very little advantage, given the cost involved in plugins handling. And the way xine was designed, it was definitely impossible to have third-party plugins developed properly.

And to finish off, I said before that plugins cannot be cleanly unloaded: the problem is not only that it’s difficult to have proper cleanup functions for plugins themselves (since often the allocated resources are stored within state variables), but also because some libraries (used as dependency) have no cleanup altogether, and they rely (erroneously) on the fact that they won’t be unloaded. And even when they know they could be unloaded, the PulseAudio libraries, for instance, have to remain loaded because there is no proper way to clean up Thread-Local Storage variables (and a re-load would be quite a problem). Which drives away another point of using plugins.

I leave the rest to you.

Zhang Le: Update on building mips64el cross toolchain

Zhang Le (r0bertz) — Sun, 08 Nov 2009 08:48:00 +0000

1. git clone git://www.gentoo-cn.org/var/git/loongson.git

2. Don't include loongson overlay's directory directly in your PORTDIR_OVERLAY. But in your own overlay, like /usr/local/portage, create symlinks pointing to sys-devel/ and sys-libs/glibc in loongson overlay. This is because you may not want to install packages from loongson overlay in your host machine. Also you need to make sure your own overlay is the first overlay containing binutils/gcc/glibc in your PORTDIR_OVERLAYS list. You may check the value of this list using command:

portageq envvar PORTDIR_OVERLAY

3. Then, just run crossdev -t mips64el-unknown-linux-gnu. This command installed the following components on my host machine:

linux-headers-2.6.27-r2 from portage.
binutils-2.19.1-r1 from loongson overlay.
gcc-4.4.2 from portage. The gcc in loongson overlay currently is just a symlink to portage gcc.
glibc-2.9_p2009042 from loongson overlay. However if you build the toolchain now, glibc-2.10.1 will be installed instead, I just updated it.

Steve Dibb: packages website going offline for a while

Steve Dibb (beandog) — Sat, 07 Nov 2009 16:35:07 +0000

I've been on a roll to clean house lately, and part of that is simplifying my hardware setup. One thing that needs to be ripped out completely is my old server, which is getting to be a real pain to maintain. Mostly it's just my personal stuff on there, but the ebuild packages website is also running on there right now. Between now and Tuesday, I'm going to take it down since I'm going to be rearranging my hardware setup anyway. I'm not going to bring the old website back online, either. The code for the new one is almost complete, and it will use a lot less resources. There's gonna bet lots of cool stuff on the new one: better feeds, simpler interface, new domain name and hopefully a new design as well. Oh, and the scripts aren't dependent upon portage anymore, which is the real crutch right now. I have to run an old version of portage (2.1.4.5) that isn't even in the tree anymore, and it's making updates painful or impossible.

The new site will also run on my dedicated Linode, where I think I've finally correctly managed the apache issues, so that means there will be less arbitrary downtime as I screw around with my box here at home. I really hate running servers at home that other people are dependent on, because I like the freedom to change things around without affecting anyone. Right now, the old site is so CPU intensive, that I can't move it over to the VPS.

The code for the new site is much cleaner. The entire thing is rewritten in OOP classes to access the portage tree, which makes my job incredibly easier. Not to mention it's a lot faster. It'll still be a bit before I get it online, but killing it will inspire me to push it along. I'm tired of having this thing limp around when it's just a dead albatross around my neck right now. So, farewell. The new one will be better.

Diego E. Pettenò: Debating future tinderbox work

Diego E. Pettenò (flameeyes) — Sat, 07 Nov 2009 14:46:36 +0000

I’ve been not working on the tinderbox lately because my “daily job” (which is not really daily) swamped me out badly. Since this week I’m going to London to take some days off, I’ll probably get back to the tinderbox after that.

For the next ride of the tinderbox, there is at least one thing that’s definitely going to be interesting: the new X11R7.5 release means that quite a bit of packages might not build at all since they don’t have the new includes fixed. I found one or two packages with such problems while doing Yamato’s root filesystem rebuild (after glibc 2.11 update).

There is another interesting idea that I should probably toy with: the way the tinderbox works, it tests all non-masked packages; by QA standards, those should not use the network at build time. During my world rebuild this night, network went offline, and one package failed since it tried to wget a piece of source from the network. And it’s not even the first one lately.

Thanks to the fact that my tinderbox uses containers I can easily isolate it out of the network so that it cannot access the network, and then make sure that the ebuilds trying to use the network get their access refused.

The other problem to cope with is the size of the logs and the fact that I still lack an analysis script and thus opening new batches of bugs requires a huge amount of work, especially when it comes to attaching the log and getting some information out of it.

Any suggestion on how to proceed with the tinderbox will definitely be welcome.

Donnie Berkholz: links for 2009-11-06

Donnie Berkholz (dberkholz) — Sat, 07 Nov 2009 01:56:11 +0000

The trough of disillusionment for Ubuntu?

“The Hype Cycle describes the way that new technologies and projects are perceived over time, if they do a good job of handling themselves, going from a technology trigger, inflated expectations, disillusionment, enlightenment, before arriving at “the plateau of productivity” – a state where there is no more hype and the new technology is simply a normal part of our lives.”

The perception over the past few years that Gentoo is dying is in reality Gentoo’s arrival at the plateau of productivity. Hype has gone away and remaining is a distribution with a true niche that fits into the broader Linux ecosystem.

(tags: gentoo communication)

Tagged: communication, gentoo, greatness, pr

Gentoo News: Gentoo KDE3 Deprecation Notice

Gentoo News () — Fri, 06 Nov 2009 06:02:08 +0000

Please turn your KDE radio on, and make sure to increase the volume to its maximum level for this important message.

After multiple setbacks we have finally managed to stabilise KDE4 on both major desktop architectures (amd64 and x86), with other teams to follow.

For this and other reasons as discussed , those of you who still use KDE3 should be seriously considering an upgrade in the near future.

The KDE3 support is being deprecated with immediate effect. This means that ebuilds are dropping KDE3 support where they were broken, or clashing with KDE4.

If you wish to still use KDE3, and you want to help others with having KDE3 around, drop a mail to kde@gentoo.org, where we can give you commit access to the special overlay which will specifically contain only KDE3 packages.
This overlay (named kde-sunset) can be easily used via layman.

Sadly upstream is not supporting KDE3 anymore and we simply lack the manpower to keep support for both (as you might have noticed in the past few months KDE3 has become more and more rusty for which we humbly apologize).

Sorry to bring you the bad news and with hope that KDE4 will suit your needs,

Tomas Chvatal
KDE Teams substituting Lead

KUDOS to Nirbheek Chauhan and David Abbott for helping to put this announcement together and Alec Warner for proofreading it.

Alex Alexander: KDE in Gentoo, an update

Alex Alexander (wired) — Thu, 05 Nov 2009 15:22:59 +0000

KDE 4.3.3 was released on Tuesday, with ebuilds instantly available to users in the main tree.

We also have KDE 4.3.1 in stable \o/

KDE 3.5 is on its way out of portage, with kde-sunset being the overlay that will (hopefully) be maintained by users still interested in it.

Finally, KDE 4.4 Beta 1 is due December 1st and you’ll find it in the kde-testing overlay as usual

Diego E. Pettenò: Distributions are the strength of Linux

Diego E. Pettenò (flameeyes) — Wed, 04 Nov 2009 20:56:18 +0000

People disagree, some people think that no operating system has any need for distributions, with all their difference and their central repositories that aren’t as central. But one of the thing that impress most the users who switch is, in many cases (at least that I could look at myself) the presence of distributions and the ability to install almost any software by simply looking it up in the package manager.

This said, when people think that overcomplex solutions are a perfect way to solve the issues that “vendors” have with distributing their software, you’re probably missing the point quite a bit. Instead of proposing changes in all the possible layers of the operating system stack, you should try to speak with with the distributors and see what you can do to make your software behave in such a way that they can lift the “send the software to the user” problem from you.

It’s a tremendously important point I’m making here: when you develop your software coming from a Windows background to work on Linux, youŕe probably making a huge amount of mistakes; the most common one is to assume that the directory to work on is the directory the program is in, or that the current working directory is the home of the user. Both differ between Windows and Linux. Fixing these minor issues is usually trivial, if you have access to the code, and if you’re willing to bend a bit around to accommodate the requests. In the case that icculus brought up, the proper solution is, generally, splitting the data from the engine, so that you can reuse the data between different architectures, and have a different engine for each architecture; or have a single huge download with all the architectures available, if they are, say, 10% over the size of the data.

The main point here is still that you have first to remember that distributions exist and that users like to rely on them (most of the time) and second to understand that neither the Windows way nor the OS X way applies to Linux. This doesn’t make Linux right and the other wrong, or vice-versa; they are three different worlds, and each one has its own good and bad side.

The biggest mistake in misunderstanding Linux for just another Windows version is providing a setup program, even worse a graphical setup program. If your software has no drivers to install, nothing to register itself into (there is no registry on Linux, after all), you most likely should not give that as the only option. First of all such a program would rarely tell you what’s going to do, and you’d also be going to run that with root privileges to install the stuff, so why should you trust proprietary software with root on your system? Of course if you’re just a “Joe User” you won’t care, you have no clue about that, but any decently skilled user would know that it’s never a good idea to trust any software you cannot control with root privileges on your box.

The second misconception is that some people seem to think that it’s a task for upstream of a project – be it a proprietary software vendor or a free software project – to provide binaries, installer and packages. This is the main reason why that silly FatELF idea is still tickling on some people. Well, let me say it once and for all it’s the distributions’ task to provide packages to the users!

Of course the problem is that distributions rarely can provide all the possible software in the world as package, may it be because their policy is to only allow Free Software (like Debian and Fedora) or for other reasons. In any case the solution is not to say “The distributions are the problem” but rather to wonder “Why are they not packaging my software?”. Of course when the problem is policy related to the license there is little to do, so you’re forced to rely on third party repositories (like RPM Fusion ) that don’t have such problems with policies. In general, a very little leeway for the distributions can go a great deal into making your software available to users.

All kind of projects who want to reach for users should listen to the distributors: that means that if a distributor complain about the way you (don’t) release software, for instance because you only use a “live” repository for the users to use, or about the way you make use of bundled libraries, you should most likely discuss with them a way to handle the situation; failing to do that is going to drive the distributor away (and then you’d probably be complaining that you’ll have to provide binaries for that distribution yourself). Unfortunately I’m quite sure that especially icculus have problems with stuff like that, given I’ve reported more than one Gentoo policy violation for ebuilds that come from icculus.

For proprietary software, this often goes not as much into the way of changing the development of the software but rather to change some distribution details: allow the developer to redistribute your software (so don’t use strange click-through download systems, don’t require the user to go a long way to find what it has to download); give a “raw tarball” option that the distribution can use as source for their packaging, be it binary packages, or source-based packages like Gentoo’s.

Move the packaging task to the packagers, they know it better.

And if you’re developing proprietary commercial software, you might want to approach some developers, and eventually give out some free licenses for them to play with so that they can package the software, and eventually give you feedback in what they would like for you to change. Most of the time, packagers are pretty pragmatic and will not be scared off by “helping proprietary software”; for instance in my overlay you can find some packaging for the Visual Paradigm Suite for which I bought a license a few weeks ago (I needed a working UML software for a job); it’s nowhere near Gentoo-ready, but I’ve not given up on it; since the Visual Paradigm customer support is also quite ready to answer to problems and suggestions, I’ve been sending them my feedback, both as user and as packager. Hopefully I might get to the point where the package is fine with Gentoo policies and I can add it to the main tree normally.

A similar situation happens with the EntropyKey software packaging since I was interested I got two of those and packaged it up; if upstream was interested in packaging this beyond their own support (I think they already have a Debian packager as part of the staff anyway), they could have created a developer program for distributors, and I’m pretty sure almost all distributors would have supported the ekeyd software in no time.

Yes, I am seeing all this situation from a packager point of view, but that’s because I definitely like this approach and instead of resent us for “not providing the stuff you want” or attacking distributions because “you have to make dozens of different packages”, try working with them. Like I said before, Ryan should stop keep inside his own little world where he can do whatever he wants and then expect people to bend at his needs, he should listen to the needs of distributors (which aren’t really so impossible!) and so should anybody who want to enter the Linux ecosystem as it is now.

And it’s definitely not only proprietary software that still doesn’t get this, Mozilla has had a hard time to get to work with distributors, OpenOffice still has such a hard time, Avidemux is a perfect example of how a package gets to ignore all the possible distribution requests (by still shipping a modified FFmpeg for instance).

Most of the time, the reasons why developers don’t want to make accommodations for distributions, are stuff along the lines of “I don’t see what difference does it make”… which is also the very reason why they have such a hard time to get their packaging together.

Steve Dibb: my blu-ray ripping trial run

Steve Dibb (beandog) — Wed, 04 Nov 2009 15:04:54 +0000

Yesterday, I wanted to see if I could rip a Blu-Ray disc using my PS3. I really want to get a BD-ROM drive, but they are so expensive still, and since I can install Linux on my PS3, I figured maybe I'd try and save myself some money and see if I could manage to get one ripped and decrypted. It actually worked, which surprised me. Ripping the disc was the simplest thing in the world, but the key on the movie I tried (Willy Wonka and the Chocolate Factory) was too new, and currently only AnyDVD has support for it. I'd love to buy a copy of that, but it only runs in Windows, and it's really expensive. Instead, I'll just have to wait for the keys to pop up eventually on the doom9 forums.

The first step, though, was getting the PS3 to run Linux. I took the shamelessly easy way out (and I don't regret it either) and installed Xubuntu. I won't go into details about how I got Linux on my PS3 since that's well documented. I will say that I remember quite vividly now why I can't stand binary distros. Bleh.

The BD filesystem is UDF. Providing you have a recent kernel (2.6.20, I think) with UDF v2.5 support, you are good to go. I mounted a remote share and just dumped the disc to an ISO file onto my desktop.

$ cat /media/cdrom0 > wonka.iso

That was the easy part.

The hard part was trying to get it decrypted. I had to use Java tools (bleh) to get to the source. There are three applications you need. And if you hate digging through forums and using download services, then I've got direct links for yah:

For Gentoo, you'll need to install the JDK to build the aacskeys library and binary. I just emerged dev-java/sun-jdk and it worked for me (I know absolutely nothing about Java, but my stabbing in the dark miraculously worked). You'll also need a runtime environment to actually execute the stuff, and I emerged dev-java/sun-jre-bin and that worked fine, too on my amd64 box.

For aacskeys and Gentoo, you'll need to apply this patch that I cobbled together from what I found on the doom9 forums to get it to compile. It just fixes the Java include directorys for the Makefile.

Now, I'm still a bit fuzzy about what each program does, and whether you need all of them or not, so I won't go into a lot of detail. What you want to use, though, is the dumphd program. But to use it, you'll need to copy the aacskeys library and a file from the bdvmdbg package as well into the path or same directory as the dumphd program.

Once you have that, you can just run dumphd.sh and it'll fire up a simple little GUI telling you if it has all the libraries it needs. Then you just specify the source and destination, and aacskeys will see if it has a working key to access the disc.

I can't really give much more detail than that, since I'm so new to this. Suffice it to say, if you read the accompanying README doc that comes with each one, you'll get along just fine.

It took me a long time last night to get just one disc ripped and transferred over my subnet to try it out, and by the time I managed to get it mounted (mount -o loop -t udf wonka.iso /mnt/udf) and access it, it was pretty late. The keys I had didn't work for my disc, and I didn't want to try the whole procedure over to try another disc.

Anyway, good luck if you try it. One thing that impressed me is how much simpler it was than I thought it'd be, but what a pain it was trying to figure out where things went wrong. The doom9 forums are a good resource, but not exactly the best place to find clear, concise information for a beginner. That part was frustrating.

Personally, I don't think it's worth the hassle right now, the way I did it. I'll get a BD-ROM sooner or later so I don't have to transfer the content over the network and can instead just test it directly. But, I started out to see if I could at least get a copy of the ISO and get the tools running all without Windows, and I can. So, that's progress right there.

Diego E. Pettenò: ELF should rather be on a diet

Diego E. Pettenò (flameeyes) — Wed, 04 Nov 2009 14:08:06 +0000

I’ve been first linked the FatELF project in late October by our very own solar; I wanted to write some commentary about it but I couldn’t find the time; today the news is that the author gave up on it after both Linux kernel and GLIBC developer dissed his idea. The post where he noted his intention to discontinue the project looks one drama-queen of a post regarding the idea of contributing to other projects… I say that because, well, it’s always going to be this way if you think about an idea, don’t discuss it before implementing, and then feel angry for the rejection when it comes. I’m pretty sure that no rejection was personal in this rejection, and I can tell you that what I would have written after reading about it the first time would have been “Nice Proof of Concept, but it’s not going to fly”.

Let’s first introduce the idea behind the project: to copy Apple’s “Universal Binaries”, that technique that allowed programs to run both on PPC-based Mac as well as the new Intel-based Mac when they decided to make the transaction, this time applying the same principle to the ELF files that are used on basically all modern UNIX and Unix-like systems (Linux, *BSD, Solaris). There is a strange list of benefits in the project’s homepage; I say strange because they really seem straw arguments for creating FatELF, since I rarely have seen this applied in real world.

Let’s be clear, when Ulrich Drepper (who’s definitely not the most charming developer in our community) says this:

Yes. It is a “solution” which adds costs in many, many places for a problem that doesn’t exist. I don’t see why people even spend a second thinking about this.

I’m not agreeing to the fact that nobody should have spent a second thinking about the idea; toying with ideas, even silly ideas like this one (because as you’ll soon see, this is a silly idea), is always worth it: it gives you an idea of how stuff works, they might actually lead somewhere, or they might simply give yo the sense of proportion of why they don’t work. But there are things to consider when doing stuff like this, and the first is that if there is a status quo, it might be worth discussing the reason of that status quo before going in full sprint and spend a huge amount of time to implement something, as the chance that’s just not going to work is quite high.

To make an example of another status quo-fiddling idea, you might remember Michael Meeks’s direct bindings for ELF files; the idea was definitely interesting, it proven quite fast as well, but it didn’t lead anywhere; Michael, and others including me, “wasted” time in testing it out, even though it was later blocked by Drepper with enough reasons and it’s no longer worked on. Let me qualify that “wasted” though: it was wasted only from the point of view of that particular feature, which led nowhere, but that particular work was what actually made me learn how the two linkers worked together, and got me interested in problems of visibility and cow as well as finding out one xine bug that would have been absolutely voodoo to me if I didn’t spend time learning about symbol resolution before.

Back to FatELF now: why do I think the idea is silly? Why am I agreeing with Drepper about the fact that it’s a solution with too high costs for the unrequested results? Well the first point to make is when Apple made the first step toward universal binaries; if you think the idea sprouted during the PPC to Intel transition, you’re wrong. As Wikipedia notes Apple’s first fat binary implementation dates back to 1994. During the M68K to PPC transition. Replicating the same procedure for an architecture change wasn’t extremely difficult to them to begin with, even though it wasn’t OSX that was used during that particular transition. The other fact is that the first Intel transition was – for their good or bad – a temporary one. As you can probably have noted, they are now transitioning from i386 software to x86-64 software (after my post on PIE you can probably guess why that’s definitely important to them).

But it goes much further than that: Apple has a long history of allowing users to port the content their computer from one to the next with each update, and at the same time they have a lot of third party providing software; since third parties started upgrading to universal binaries before Intel Macs were released for the users, if users kept up to date with the release, one they got their new Intel Mac they would just had to copy the content from the old to the new system and be done with it. This is definitely due to the target audience of Apple.

There is another thing to know about Apple and OS X, that you might not know about if you’ve never used a Mac: applications are distributed in bundles, which are nothing more than a directory structure, inside which the actual binary is hidden; inside the bundle you find all the resources that are needed for the program to run (translations, pictures, help files, generic data files, and so on). To copy an application you only have to copy the bundle, to remove almost all applications you just shove the bundle in the trash can. This forces distributions to happen in bundles as well, which is why Universal Binaries were so important to Apple: the same bundle had to work for all people so that it could still be copied identical between one computer to the other and work, no matter the architecture. This is also why, comparing the size of bundles built Universal, PPC-only and Intel-only, the first is not as big as the size of the other two: all the external resources are shared.

So back to Linux, and see how this applies: with a single notable exception all the Linux distributions out there use a more or less standard Filesystem Hierarchy Standard compatible layout (some use LSB-compatible layout, the two are not one and the same, but the whole idea is definitely similar). In such a setup, there are no bundles, and the executable code is already separated from the code that is not architecture-dependent (/usr/share) and thus shareable. So the only parts that cannot be shared, that FatELF would allow to be shared are the executable code parts, like /bin and /lib.

Now let’s start with understanding where the whole idea is going to be applied: first of all, Linux distributions, by their own design, have a central repository for software, which OS X does not have; and that central repository can be set up at installation time for getting the correct version of the software, without asking the user to know about the architecture by itself. The idea of using fat binaries to reduce the size of that repository is moot: the shareable data is already, for most distributions I know, shared in -noarch packages (arch-independent); the only thing you’d be able to spare would be the metadata of packages, which I’m quite sure for most “big” applications is not going to be that important. And on the other hand, the space you’d be saving on the repository side is going to be wasted by the users on their harddrive (which is definitely going to be disproportionally smaller) and by the bandwidth used to push the data around (hey, if even Google is trying to reduce the downloaded size fatelf is not only going against the status quo but also the technical trend!).

And while I’m quite sure people are going to say that once again, disk space is cheap nowadays, and thus throwing more disks at the problem is going to fix it, there is one place where it’s quite difficult to throw more space at it: CDs and DVDs, which is actually one of the things that FatELF is proposing to make easier, probably in light of users not knowing whether their architecture is x86, amd64 or whatever else. Well, this is already been tackled by projects such as SysRescueCD that provide two kernels and a single userland for the two architectures, given that x86-64 can run x86 code.

The benefits listed in FatELF’s page seem also to focus somewhat to the transition between one arch and the other, like it’s now happening between x86 and x86-64; sure it looks like a big transition and quite a few players in the market are striving to do their best to make the thing as smooth as possible, but either we start thinking of the new x86-64 as the arch, and keep x86 as legacy, or we’re going to get stuck in a transition state forever; Universal Binaries for Apple played a fundamental role in what has been a temporary transition, and one they actually completed quite fast: Snow Leopard does no longer support PPC systems, and everybody is expected the next iteration (10.7) to drop support for 32-bit Intel processors entirely to make the best use of the new 64-bit capabilities. Sure there could be some better handling of transitioning between architectures in Linux as well, especially for people migrating from one system to the other, but given the way distributions work, it’s much easier for a new install to pick up the home directories set up in the older system, import the configuration, and then install the same packages that are installed in the previous one.

After all, FatELF is a trade-off: you trade bigger binaries for almost-universal compatibility. But is the space the only problem at stake here? Not at all; to support something like FatELF you need changes at a high number of layers; the same project page shows that changes were needed in the Linux kernel, the C library (glibc only, but Linux supports uclibc as well), binutils, gdb, elfutils and so on. For interpreted language bindings you also have to count in changing the way Ruby, Python, Java, and the others load their libraries since they now hardcode the architecture information in the path.

Now, let’s get to the real only speakable benefit in that page:

A download that is largely data and not executable code, such as a large video game, doesn’t need to use disproportionate amounts of disk space and bandwidth to supply builds for multiple architectures. Just supply one, with a slightly larger binary with the otherwise unchanged hundreds of megabytes of data.

You might or might know that icculus.org where the FatELF project is hosted is the home of the Linux port of Quake and other similar games, so this is likely the only real problem that was, up to now, really come up before: having big packages for multiple arches that consists mostly of shareable data. As said before, distributions already have architecture-independent packages most of the time; it’s also not uncommon for games to separate the data from the engine source itself, since the engine is much more likely to change than the data (and at the same time, if you use the source version you still need the same data as the binary version). The easiest solution is thus to detach the engine from the data and get the two downloaded separately; I wonder what the issue is with that.

On the other hand, there is a much easier way to handle all this: ship multiple separate ELF binaries in the same binary package, then add a simple sh script that calls the right one for the current host. This is quite easy to do, and requires no change at any of the previously-noted layers. Of course, there is another point made on the FatELF project page that this does not work with libraries… but it’s really not that of an issue, since the script can also set LD_LIBRARY_PATH to point to the correct path for the current architecture as well. Again, this would solve the same exact problem for vendors without requiring any change at all in the layers of the operating system. It’s transparent, it’s easy, it’s perfectly feasible.

I hear already people complaining “but a single FatELF file would be smaller than multiple ELF files!”. Not really. What you can share between the different ELF objects, in theory, is still metadata only (and I’m not convinced by the project page alone that that’s what it’s going to do, it seems to me like it’s a sheer bundling of files together): SONAME, NEEDED entries and similar. Unless you also start bundling different operating systems together – which is what the project also seem to hint at – because in that case you also have no warranty that the metadata is going to be the same: the same code will require different libraries depending on the operating system it’s built for.

Generally, an ELF file is composed of executable code, data, metadata related to the ELF file itself, and then metadata related to the executable code (symbol tables, debugging information) and metadata related to the data (relocations). You can barely share the file’s metadata between architectures, you definitely cannot share it between operating systems as stated above (different SONAME rules, different NEEDED).

You could share string data, since that’s actually the same between different architectures and operating systems most of the time but that’s not really a good reason; you cannot share constant data because there are different ordering, different sizes and different paddings across architectures, even two very alike like x86 and x86-64 (which is why it’s basically impossible to have inter-ABI calls!).

You cannot share debugging information either (which might be the big part of an ELF file) because it’s tied to the offset of the executable code, and the same applies to the symbol tables.

So, bottomline, since there are quite a few strawy benefits on the FatELF project page, here is a list of problems caused by that approach:

introduces a non-trivial amount of new code at various layers of the system (kernel, loader, linker, compiler, debugger, language interpreters, …), it doesn’t matter that a lot of that code is already published by now, it has to be maintained long-time as well, and this introduces a huge amount of complexity;
would increase dramatically the size of downloading packages for the optimistic case (a single architecture throughout a household or organisation) since each package would comprise of multiple architectures at once;
would use up more space on disk since each executable and library would then be duplicated entirely multiple times; note that at the time Universal Binaries started popping up on systems, more than one software was released to strip the other architecture out of them to reduce space to be wasted on already-ported or won’t-be-ported systems; while FatELF obviously comes with the utilities by itself, I’m pretty sure most tech-savvy users would then decide simply to strip off the architectures that are useless to them;
would require non-trivial cross-compilation from build servers which right now all the distributions, as far as I know, tend to avoid.

In general, distributions will definitely never going to want to use this; free software projects would probably employ better their time by making sure the software is easily available in distributions (which often means they should talk to distributors to make sure their software has an usable build system and runtime configuration); proprietary software vendors might be interested in something like that – if they are insane or know nothing about ELF, that is – but even then the whole stack of changes needed is going to be way disproportionate to the advantages/

So I’m sorry if Ryan feels bad about contributing to other projects now because people turned down his idea, but maybe he should try for once to get out of his little world and see how things work with other projects involved, like discussing stuff first, asking around and proposing: people would have turned him down with probably most of the same arguments I used here today, without him having to spend time writing unused (and unusable) code.

Steve Dibb: closed captioning on dvds (and ripping them)

Steve Dibb (beandog) — Tue, 03 Nov 2009 21:10:38 +0000

In ripping my DVDs, I try to future-proof it as much as I can, by putting in as many elements as I *think* I might need or want someday down the road. One of those elements is subtitles. There are three types of subtitles that can be on DVDs -- VobSub, closed captioning and SDH -- and the first two can be extracted fairly easily. I have no idea how to access the SDH ones. I think you need either a newer DVD player or a Blu-Ray one.

I've been ripping my TV shows, and so far I haven't seen any really hard and fast rules on what to expect with them on DVD. Part of the reason is that I just haven't been paying much attention to subtitles until recently.

I was playing with ripping one show last night, and I saw the CC logo on the back of the case, so I went to check the rest of my library to see which other ones had it. Nearly my entire library of Warner Bros. DVDs displayed the logo -- even for much older cartoons (Looney Tunes, Scooby Doo) -- once again staying consistent with the fact that the studio puts a lot of effort into the quality of their releases.

I just started playing with extracting CC though, and just barely wrote the code to my DVD ripper to extract them, so I have no idea what the other series are like, if they have subtitles or not -- VobSub or CC. I usually don't find out until I actually go to rip them.

Extracting the closed captioning subtitles is a lot easier and faster than getting the VobSub streams. For Linux (and Mac and Windows) there's a nifty OSS program called ccextractor. Once you have your VOB video file on your harddrive, just run that on the movie, and it will create an SRT subtitle file of the closed captioning text. It's great, and really fast, taking probably under a minute on a 60-minute video on my box. Comparatively, when ripping a VobSub stream, you need to read the DVD directly which causes its own bottleneck, and then demux the entire stream. It takes probably around 3 to 5 minutes for an episode of the same length.

Another thing I like about the closed captioning titles is that because they are extracted as SRT, it's easy to look through them since they are just text files. If you're really anal, you can correct typos yourself. The VobSub subtitles are all bitmaps. I've also noticed that on some DVDs, where there were issues with framerates or something else, that the VobSub timestamps will be off ... and sometimes either they will show up clumped together at the beginning of the film or the sync will be way off. I think that this has to do with the dumping process, somewhere, but I'm not sure. I've never really taken the time to pin down the source.

So, with closed captioning being easier and faster to extract, as well as editable and the timestamps haven't had any issues for me (yet), it's quickly becoming my preferred subtitle format.

There's only one small issue with using ccextractor, and that is you won't know if there are any captions in the VOB until after it's made its trial run. The program will create an .srt file regardless when you run it, but the file will be empty if it couldn't find any. That's the only drawback. With VobSub, you can know if there are subtitles just by probing the DVD using lsdvd or something similar.

Muxing it into matroska is simple, too. Just pass it as a file argument and you're done.

As a sidenote, while my bend application that I wrote and use to rip DVDs would be a major pain to setup for someone else, I've rewritten it recently so that it uses individual classes to access every object directly: DVD, DVD track, DVD VOB, Matroska file. They are standalone classes written in PHP if anyone wanted to use them, feel free. You would also need my tiny class of shell functions as well, since they all make calls to it.

The DVDVOB one makes it simple to extract the subtitle stream. In fact, all the classes make things relatively simple. They have made writing my code so much simpler.

Diego E. Pettenò: The PIE is not exactly a lie…

Diego E. Pettenò (flameeyes) — Mon, 02 Nov 2009 16:33:52 +0000

One very interesting misconception related to Gentoo, and especially the hardened sub-profile, is related to the PIE (Position-Independent Executable) support. This is probably due to the fact that up to now the hardened profile always contained PIE support, and since it relates directly to PIC (Position-Independent Code) and PIC as well is tied back to hardened support, people tend to confuse what technique is used for what scope.

Let’s start with remembering that PIC is a compilation option that produces the so-called relocatable code; that is, code that is valid no matter what base address it is loaded at. This is a particularly important feature for shared objects: to be able to be loaded by any executable and still share the code pages in memory, the code needs to be relocatable; if it’s not, a text relocation has to happen.

Relocating the “text” means changing the executable code segment so that the absolute addresses (of both functions and data — variables and constants) is correct for the base address the segment was loaded at. Doing this, causes a Copy-on-Write for the executable area, which among other things, wastes memory (each process running will have to have its private copy of the executable memory area, as well as the variable data memory area). This is the reason why shared objects in almost any modern distribution are built relocatable: faster load time, and reduced memory consumption, at the cost of sacrificing a register.

An important note here: sacrificing a register, which is something needed for PIC to keep the base address of the loaded segment, is a minuscule loss for most architectures, with the notable exception of x86, where there are very few general registers to use. This means that while PIC code is slightly (but not notably) slower for any other architecture, it is a particularly heavy hit on x86, especially for register-hungry code like multimedia libraries. For this reason, shared objects on x86 might still be built without PIC enabled, at the cost of load time and memory, while for most other architectures, the linker will refuse to produce a shared object if the object files are not built with PIC.

Up to now, I said nothing about hardened at all, so let me introduce the first relation between hardened and PIC: it’s called PaX in Linux, but the same concept is called W^X (Write xor eXecute) in OpenBSD – which is probably a very descriptive name for a programmer – NX (No eXecution) in CPUs, and DEP (Data Execution Prevention) in Windows. To put it in layman terms, what all these technologies do is more or less the same: they make sure that once a memory page is loaded with executable code, it cannot be modified, and vice-versa that a page that can be modified cannot be executed. This is, like most of the features of Gentoo Hardened, a mitigation strategy, that limits the effects of buffer overflows in software.

For NX to be useful, you need to make sure that all the executable memory pages are loaded and set in stone right away; this makes text relocation impossible (since they consists of editing the executable pages to change the absolute addresses), and also hinders some other techniques, such as Just-In-Time (JIT) optimisation, where executable code is created at runtime from an higher, more abstract language (both Java and Mono use this technique), and C nested functions (or at least the current GCC implementation, that makes use of trampolines, and thus require executable stack).

Does any of this mean that you need PIC-compiled executables (which is what PIE is) to make use of PaX/NX? Not at all. In Linux, by default, all executables are loaded at the same base address, so once the code is built, it doesn’t have to be relocated at all. This also helps optimising the code for the base case of no shared object used, as that’s not going to have to deal with PIC-related problems at all (see this old post for more detailed information about the issue).

But in the previous paragraph I did write some clue as to what the PIE technique is all about; as I said, the reason why PIE is not necessary is that by default all executables are loaded at the same address; but if they weren’t, then they’d be needing either text relocations or PIC (PIE), wouldn’t they? That’s the reason why PIE exists indeed. Now, the next question would be, how does PIE relate to hardened? Why does the hardened toolchain use PIE? Does using it make it magically possible to have a hardened system?

Once again, no, it’s not that easy. PIE is not, by itself, neither a security measure nor a mitigation strategy. It is, instead, a requirement for the combined use of two mitigation strategy, the first is the above-described NX idea (which rules out the idea of using text relocations entirely), while the second is is ASLR (Address Space Layout Randomization). To put this technique also in layman terms, you should consider that a lot of exploit require that you change the address a variable points to, so you need to know both the address of that variable, and the address to point it to; to find this stuff out, you can usually try and try again until you find the magic values, but if you randomize the addresses where code and data are loaded each time, you make it much harder for the attacker to guess them.

I’m pretty sure somebody here is already ready to comment that ASLR is not a 100% safe security measure, and that’s absolutely right. Indeed here we have to make some notes as to which situation this really works out decently: local command exploits. When attacking a server, you’re already left to guess the addresses (since you don’t know which of many possible variants of the same executable the server is using; two Gentoo servers rarely have the same executable either, since they are rebuilt on a case by case basis — and sometimes even with the same exact settings, the different build time might cause different addresses to be used); and at the same time, ASLR only changes the addresses between two executions of the same program: unless the server uses spawned (not cloned!) processes, like inetd does (or rather did), then the address space between two requests on the same server will be just the same (as long as the server doesn’t get restarted).

At any rate, when using ASLR, the executables are no longer loaded all at the same address, so you either have to relocate the text (which is denied by NX) or you’ve got to use PIE, to make sure that the addresses are all relative to the specified base address. Of course, this also means that, at that point, all the code is going to be PIC, losing a register, and thus slowed down (a very good reason to use x86-64 instead of x86, even on systems with less than 4GiB of RAM).

Bottomline of the explanation: using the PIE component of the hardened toolchain is only useful when you have ASLR enabled, as that’s the reason why the whole hardened profile uses PIE. Without ASLR, you will have no benefit in using PIE, but you’ll have quite a few drawbacks (especially on the old x86 architecture) due to building everything PIC. And this is also the same reason why software that enables PIE by itself (even conditionally), like KDE 3, is doing silly stuff for most user systems.

And to make it even more clear: if you’re not using hardened-sources as your kernel, PIE will not be useful. This goes for vanilla, gentoo, xen, vserver sources all the same. (I’m sincerely not sure how this behave when using Linux containers and hardened sources).

If you liked this explanation that costed me some three days worth of time to write, I’m happy to receive appreciation tokens — yes this is a shameless plug, but it’s also to remind you that stuff like this is the reason why I don’t write structured documentation and stick to simple, short and to the point blogs.

Diego E. Pettenò: Changes in the comment antispam

Diego E. Pettenò (flameeyes) — Sat, 31 Oct 2009 11:02:01 +0000

You probably remember the series of posts I already wrote about my antispam that uses the User-agent field to reject at the source a number of comments that are likely to be spam. The idea is definitely working right, just yesterday it filtered out 134 spam comments (no false positives, after a quick check), and at the same time I have no need to use obnoxious captchas, or to block comments on old posts (and just yesterday I got an interesting one on an almost year old post ).

Unfortunately this was still not perfect; luckily there is a second antispam pass that is applied directly by Typo using some heuristics (like the number of links) and akismet; this second pass is both good and bad. For instance it always marks as spam the posts where people do provide references for their comment, which is a bit tiresome. Sure it does not delete the posts, but only queue them up for moderation, but still. Unfortunately the second pass couldn’t be disabled or loosened up because usually I would get around three spam comments every day or so (which is still a lot less compared to the hundreds sometimes the filter kills at the source).

But last night, thanks to Mark, I was able to refine the antispam even more (and the comment policy now is updated to reflect that); I added a couple more DNSBL (DNS-based blacklists): proxyBL DroneBL and CBL . I left them running on the untested input during the night and the results are quite interesting. Just one or two hits on ProxyBL, but about two posts an hour hit DroneBL right away, and of those a few wouldn’t have hit my usual User-Agent-only antispam.

But since I don’t want to hit other services when I can filter the spam myself, I’ve now re-configured the checks to only apply if the comment didn’t hit any other check first (this way all the bogus user agent posts would be dropped and then the remaining “valid” ones would be checked). In particular, CBL is set as the very last check, for a very important reason: CBL does not sanction its use for non-mail related filtering. Unfortunately, CBL is also the only list that had a couple of IP addresses from which false negatives arrived yesterday, so I really wouldn’t have wanted to ignore it entirely. But I am responsible for any problem related to CBL with this kind of use; please don’t ever bother CBL upstream about this.

And another change, related to the blog spam, might be of interest. I’ve tried re-enabling the trackback support, but as it was easy to guess, there seems to be nothing but spam passing through it nowadays; very few valid installations actually use the trackback support, and they definitely don’t justify the amount of spam I’d be getting; on the other hand, Typo should be able to trackback itself to link posts together when I note something about them, and that’s one thing that I’d really like to keep; so for now I’ve enabled the trackback feature from within Typo, but I’ve stopped it on the Apache configuration, by allowing only the server’s own IP address to access the location.

I’ll publish the modsecurity configuration someday in the near future, hopefully.

Damien Krotkine: Curses::Toolkit is on CPAN

Damien Krotkine (dams) — Fri, 30 Oct 2009 23:30:52 +0000

After few months of delay, Curses::Toolkit is now available on CPAN.

Curses::Toolkit tries to be a modern toolkit to implement semi-graphical interfaces using Curses. It's inspired by Gtk.

I gave a presentation of Curses::Toolkit at the French Perl Workshop 2009, and at the Open Source Developers conference France 2009. BooK presented it as part of a Lightning Talk at YAPC EU 2009 where I couldn't participate. Thanks to him for that. I will try to participate to London Perl Workshop this year and give a talk about it as well. I have received quite good feedback from people and users about it.

I'd like to mention that Jerome Quelin helped me with the initial release, injecting some Dist::Zilla magic into Curses::Toolkit. Thanks, Jerome.

What next ? I'll try to post a video of what is currently possible with Curses::Toolkit. I need to add more widgets and signals, and fix some bugs. Also, migrating to Moose is on the todo list. The module is usable now, but you can't really build a very nice big application with it.

Want to help ? If you have some free time, please consider helping me on next development of the module. You don't need to be a coder, here are some of the things you could help with :

Improve documentation : current documentation has some errors, and a better front page for the module would be good.
Create a Tutorial : preferably in POD, but not tied to follow sub-modules' names
Help implementing additional widgets
Create new themes : it's all about colors
Help the migration to Moose

If you're interested, contact me. The source is here

Jeremy Olexa: Gentoo: About “optimizing”

Jeremy Olexa (darkside) — Fri, 30 Oct 2009 18:26:29 +0000

As Linux-Mag points out (Gentoo Optimizations Benchmarked) using gcc optimizations for "omg, teh speed" is not all that practical. Sure, I'll add some compiler flags here and there as long as I am compiling everything anyway but I don't consider that a feature of Gentoo Linux.

I actually prefer Gentoo for the package management and customization via USE flags (even with the headaches that they cause sometimes).

Doug Goldstein: Improved VDPAU abstraction

Doug Goldstein (cardoe) — Fri, 30 Oct 2009 13:47:52 +0000

Recently, Aaron Plattner from NVIDIA announce libvdpau 0.2, which a wrapper for driver specific VDPAU implementations. You may have noticed that nvidia-drivers shipped a libvdpau.so library for a while now for apps to link against while placing their actual implementation in libvdpau_nvidia.so. Newer ebuilds of nvidia-drivers will no longer install libvdpau.so, nor any of the headers and install will rely on this being installed. Other applications, like MythTV will instead depend on x11-libs/libvdpau.

In addition to this change, there’s a new package called vdpauinfo in the tree. Some people might know it from the NVNews forums as vdpinfo. However, Aaron got the author’s permission to add it to freedesktop.org and rename it to vdpauinfo to match the library.

Doug Goldstein: nvidia-drivers and xorg-server compatiblity

Doug Goldstein (cardoe) — Fri, 30 Oct 2009 03:42:28 +0000

As always happens when we see a new xorg-server release people start to wonder why the Gentoo ebuilds for nvidia-drivers have blockers for the new release. To help clarify this I’ll post an ASCII table showing the current support.

driver series   maximum ABI
-------------   -----------
71.86.*         2.x
96.43.*         5.x
173.14.*        5.x
190.*           6.x

xorg server     video ABI
-----------     ---------
1.7.*           6.x
1.6.*           5.x
1.5.*           2.x

As you can see from the above, the 71.86.x series has gone into very limited maintenance mode. I would be surprised to see very many updates, if any at all to it in the future. However, for users of the 96.43.x and 173.14.x series, NVIDIA has confirmed to me that there will be an update to the latest xorg-server ABI change. This update though does not have any definite dates so I can’t provide any. I recommend users of that series stay with xorg-server-1.6.x releases for the time being so they can have a smooth X experience.

Jeremy Olexa: Using sshfs with rtorrent

Jeremy Olexa (darkside) — Fri, 30 Oct 2009 01:56:31 +0000

I had this genius idea about using sshfs with rtorrent. I thought that this use case would fit best in situations where you have good bandwidth but not much diskspace, such as my linode VPS (review). So, I'll attempt to share my findings in this regard.

If you are not familiar with rtorrent. You just need to know that it is a powerful, lightweight bittorrent client. It has a "watch" feature that watches a directory for new torrents, and obviously it can put downloaded files in a specified location. I tried both of these with sshfs.

First, I was having trouble with rtorrent just 'freezing' up when I put a torrent file in the sshfs accessible watch dir. I didn't quite know what was wrong here. Research led me to rtorrent bug 322 and that sshfs did not support filesystems without mmap properly. Darn. More research led me to a recent kernel commit that looked promising. Low and behold, reboot my host with 2.6.31.x kernel and rtorrent works with sshfs watch and destination directory. Yay.

Well, not so fast...

The performance is quite poor with the destination directory on sshfs. This is to be expected because now your download speed for torrents is limited to the download speed of your final destination. But, rtorrent was only giving me a sustained speed of 1/4 of that demonstrated with a simple file copy to the destination. I speculate that this is from the rtorrent overhead or maybe fragmenting? Not sure exactly and I don't care. My solution to this was to use the rtorrent "move on finished" feature that downloads the file to local disk and then moves it to sshfs destination after it is finished. Amazingly, this works quite well.

My testing scenario was the following:
-79MB Gentoo 2008.0 install cd torrent. With the complete sshfs solution, it took ~6 minutes to download (to the sshfs destination) and then 5 minutes to check the hash. So, roundtrip of 11 minutes from start download to seeding. With the on_finished solution, it took 1 minute to download (to local disk) and 1 minute to check the hash and move to the sshfs destination. For a roundtrip of ~2 minutes from start of download to seeding.

In conclusion, this isn't the perfect solution because you impose a large bottleneck into the mix and unintended I/O activity on the local disk. However, it works for me and what I am doing. Maybe it will give someone else some ideas in the future.

Gilles Dartiguelongue: Gnome 2.28.1 full steam

Gilles Dartiguelongue (eva) — Fri, 30 Oct 2009 00:04:29 +0000

http://cia.vc/stats/author/eva

My CIA profile went from about one commit every 17.35 housr to one every 16.56. The difference does not seem big but the calculation is diluted on about 6 years due to a KDE dev sharing the same nick. This amounts to 158 commits tonight.

~arch is now at about 85% of completeness for Gnome 2.28.1, a few commits are missing due extra complexity (hey it still took my 3 hours to do that). Beware that this release still has a few rough edges, especially policykit migration buts. So if you get cut, please come to bugzilla but do not expect sweet words and attention if I see comments like "dude why do you keep on breaking ~arch". It's ~arch, beat it.

Original post blogged on b2evolution.

Damien Krotkine: New Home

Damien Krotkine (dams) — Thu, 29 Oct 2009 21:39:19 +0000

The new server I had setup few weeks ago got hacked, brute force ssh attacks being made from it... I had spent some time setting it up, and now I'm bored.

So, I decided to migrate my blog to a hosted solution. As I'm a big fan of movable type, I decided to give Typepad a go. So this is yet another rebirth of my blog (I think it's the 6th time now).

I'll use that opportunity to catch up with the Perl blogging Iron Man competition.

More to come about Perl and Curses::Toolkit, my baby :)

David Abbott: Podcast 65 Bashed In The Head

David Abbott (dabbott) — Thu, 29 Oct 2009 18:33:43 +0000

In this podcast I talk about my progress learning Bash for Gentoo ebuild development.

LINKS:
Bash By Example
http://www.ibm.com/developerworks/library/l-bash3.html
Linux Command Line and Shell Scripting Bible
http://xrl.us/bfzigo
Learning the Bash Shell
http://oreilly.com/catalog/9780596009656
Gentoo Tools
http://devmanual.gentoo.org/tools-reference/index.html
Sed By Example
http://www.gentoo.org/doc/en/articles/l-sed1.xml
Bash Guide
http://mywiki.wooledge.org/BashGuide
Grep
http://www.panix.com/~elflord/unix/grep.html
Awk
http://www.grymoire.com/Unix/Awk.html

irc network freenode channel #linuxcrazy

Download

Remi Cardona: Response to a comment

Remi Cardona (remi) — Thu, 29 Oct 2009 08:30:25 +0000

Benjamin wrote a comment on my last post, and I'll share my answers here because those questions come up every now and then, so it's better to try to inform everyone. (That and I never write on this blog, so this is a perfect excuse to do so)

If you assume compile problems, why is that thing unmasked?

Xorg-server 1.7 is not getting stabilized, it's just getting unleashed onto unstable. Unstable means exactly that. Of course we try to do our best and we won't release something we know will break. The idea behind unstable is for users to test the new and shiny stuff before it hits stable.

If you don't want to help fix bugs, use stable. It's as simple as that.

I've always been irritated by the way the xorg team handled masked/unstable/stable releases, as even rc's were unmasked at times.

Releases in X-land are tough. The numbers almost mean nothing. For instance, the last stable version in the 1.5 series was 1.5.3-r6. And despite the apparently stable version number, it currently has 80 patches to make it run smoothly.

On the opposite side, the current stable server is 1.6.3.901-r2, which is indeed a "pre point release" only has a couple patches. And 1.7.1 doesn't have any patches.

So don't let the version number fool you, they mean almost nothing.

As for what we put in portage, well X is a complex piece of software. It used to have more than a million lines of code and it's been getting some tough love these last 2 or 3 years. And up until recently, drivers were a mess. I had shivers every time a new driver was released : "How many systems will this break?" was a question I asked myself over and over.

There are probably a lot of people who put the xorg-server in package.keywords because they needed/wanted feature X/Y or because it fixed some bug for them (it did for me). So now I get a release that possibly breaks build in unstable?

Again, unstable is for power users who are not afraid of filing bug reports if something breaks. We try to make sure that things don't break every day, but Gentoo being a source distro with billions of possibilities (USE flags, CFLAGS, arches, packages, ...),you can't reasonably expect us to try every possible combination.

So we ask for you help (via bugzilla) in return. Gentoo is a community distro, after all.

So there, that's it for today, I hope y'all know a bit more about how we manage X and unstable packages.

Doug Goldstein: Subversion Branch Office Support aka commit-redir

Doug Goldstein (cardoe) — Thu, 29 Oct 2009 05:45:19 +0000

Right now at work we’re currently having an issue with branch offices and SVN. We’ve got a machine with some decent power with decent disk space handling our SVN repos. We’re running a 1.6 version repo that was dumped and reloaded with 1.6 so its using the newer format fully. However we still have employees at branch offices that often complain about the poor speed of SVN. Oddly the biggest complainers were Windows based devs and people using obscure (at least to me) GUI SVN tools. Currently our branch offices are all linked back to our main office over a VPN link with a dedicated 1.5mbit up and down to each remote office, with a possibility to burst up to 80% of total VPN bandwidth. A few complaints were attributed to poorly coded SVN clients (i.e. one employee’s client would perform an ‘svn log’ on the top level of the repo and filter out the correct level of messages on the client side). However the complaints have remained constant. To mitigate our guy handling our infrastructure configured SVN hotcopy repos at the branch office servers that allow everyone to get their data off those SVN servers (repo UUIDs were sync’d) and then commit back to the main office.

To remedy this I’ve created a new capability in SVN trunk. I’ve called it ‘commit-redir’. Basically a branch office employee can now checkout their code from a local SVN repo and commit as they normally would to that server instead of having to go through the time consuming steps of switching back to the main office repo. Transparent to them, the client advertises the ‘commit-redir’ capability to their local server, which then sees that its a read only mirror and replies back with the correct URL they should present the commit to. The client takes this response and re-submits the commit back to the main office. Net result, employees can enjoy the full speed of having a local mirror of SVN except when committing. They don’t have to jump through loops to use that local mirror and commit back.

Now I’ve still got some bugs in the code and some touch ups to do. We’re also not actually running the code at the office yet due to the afore mentioned bugs but I’m hoping this week to submit the code upstream and to start running it at our offices.

Doug Goldstein: nvidia-drivers 190.x

Doug Goldstein (cardoe) — Thu, 29 Oct 2009 03:23:19 +0000

NVIDIA has officially blessed the 190.x driver series to replace the 185.18.x series. Release highlights include:

xorg-server-1.7 support
OpenGL 3.2 support
VDPAU decoding of MPEG-4 Part 2, DivX 4, and DivX 5 depending on your GPU
Support additional GPUs in the GeForce GT series

For more details read their release notes, here.

Gentoo has had a few different developers maintaining “unofficial” drivers in the 190.x series. However there’s a few outstanding issues with those ebuilds that still need to be resolved before they’re unmasked. I’ll hopefully push 190.42-r2 unmasked later on this evening.

Diego E. Pettenò: Blog posts are no replacement for documentation

Diego E. Pettenò (flameeyes) — Tue, 27 Oct 2009 22:23:56 +0000

Since I was asked in a previous post I’d like to make some notes about why I “document by blog post” in so many occasions.

I know perfectly well that my blog posts are no replacement for proper documentation; code, procedures and policies need to be properly documented, and tied to the project they are supposed to document. Documentation by blog post is difficult to write, manage and search, and can be indeed useless for the most art.

So why do I write it? Well, most of the time I start a blog post with some ideas in mind, write down it, and then depending on the feedback I either continue the topic or drop it entirely. I guess the most prominent counter-example is the For A Parallel World (which I know I haven’t updated in a while).

Writing proper documentation is important, and I know that pretty well, I have written and ranted about that before as well. And it’s knowing that, that I started the Autotools Mythbuster project which, to be honest, has given me mixed feedback, and satisfaction. The problem is: writing a blog takes just a modicum of effort, because I don’t have any obligation about form, or grammar, or language; I might soft-swear from time to time in a post, I might rant, I might have some smaller mistakes around, both in grammar and content, and so on. I don’t go updating blog posts to fix grammar and style and so on. Writing complex and organized documentation requires a lot more work, and when I say a lot I mean quite a lot more. Of course the result is also of much higher quality, because of that.

I have tried finding alternative routes to get the good results out without having to just apply that much effort in my (unpaid) free time; the first option was LWN, which actually helped me paying for a good part of Yamato’s hardware. Unfortunately LWN is not a perfect solution for me; partly because my topics tend to be quite low-level, too low-level for the LWN readers I’m afraid, and too distant from the Kernel as well (which is probably the only low-level area that LWN really writes a lot about); the other problem is that LWN is still something similar to a magazine, a journal, and thus does not allow an easy way to organised documentation; like autotools-mythbuster is. It would still be a puzzle of entries; of higher quality than a blog, but still a puzzle.

The classical form for organised documentation is that of a book; in today’s age, ebooks are also quite often used, to avoid the whole mass-production and distribution trouble for topics that might not be of enough interest (interestingly enough, that’s not true still for a lot of books, so lately I actually had to by more paper books because I couldn’t find PDFs of them to use with the Reader). Now, this also have troubles; as you might remember I already tried looking for a publisher for Autotools Mythbuster, before going with the open project it’s now.

The idea behind that would have been putting as much effort as possible into that single piece of documentation, complete it as much as possible and get it out in some complete form. There you go: high-quality results, paid effort, and organised up. Unfortunately, finding a publisher is never an easy task, and for that topic in particular, I ended up hitting a stone wall: O’Reilly already had somebody working on the topic, and the book is out now I think (I haven’t read it). This actually was ignoring a problem with classical books: they cannot easily be updated; and documentation often has to be, to correct mistakes, grammar, style, and especially to be kept up to date with what they document. For instance, Autotools Mythbuster has a specific section on forward porting (which I’ll probably keep updating for the future versions as well).

So the final option was making it an open book; again, the effort is not ignorable, so my first solution was to write on it on a donation basis: would have covered the effort I needed to put into it, and would still have been able to be there for everybody. I didn’t count in the fact that the topic is too developer-oriented to actually be of any use to people who would be donating. Indeed, I wish to thank the last two donors (in terms of time), Thomas Egger (who sent me a good mouse to replace the stupid Mighty Mouse, you’ll soon see results about that, by the way), and Joseph Booker (who sent me some books, I started with The Brief Wondrous Life of OScar Wao because I was meaning to read it for almost two years now, but the useful one will soon prove useful, I’m sure). But they, like most others, never explicitly named the guide. And so I’m trying to find more time for the general postings than that in particular.

Just a note before you start wondering about the guide; yes I haven’t updated it in a while. Why? Because I sincerely feel like it’s not useful any more. As I said it requires a positive amount of effort to be extended; there is, true, some interest on it, but not enough to actually have moved anyone to ever try funding its extension. With O’Reilly now publishing a complete book on the matter, I don’t think it’s worth my time keeping it up. I might still extend it if I have to correct some build system, or if I discover something new, but not going to keep extending it by my own will without such a need.

Bottom-line: I could probably write more extensive, organised, and precise documentation about lots of stuff, especially the stuff I write about on the blog from time to time, but the problem is always the same: it requires time and effort; and both are precious commodity; most of my time is already committed to paid work nowadays, and Gentoo is getting more and more to the third place (first is work, second health). Documenting what I can with the blog is, in my opinion, still better than nothing, so I’ll keep doing that.

Josh Saddler: Intel graphics and gaming, Abiword 2.8.0

Josh Saddler (nightmorph) — Tue, 27 Oct 2009 19:07:03 +0000

Last night I installed UT2004 on my laptop, after not playing it since June. The laptop in question is an older ThinkPad R61i, with an Intel X3100 graphics chip. I know -- not the best for gaming. However, most online reports I found indicate that it's acceptable for such an old game as UT2004, so I figured it'd be worth a shot. The Intel graphics drivers have made a lot of progress in the last two years, especially on the 3D front, right? Right?

Kinda. After reducing all settings to "low" and dialing back the resolution to 1024x768 (native is 1280x800), the game is playable, but with very uneven framerates. Looking toward the middle of a map, or anyplace with a lot of action, introduces a good stutterfest; frames are down to between 8 and 18FPS. I enabled a few extra options such as pixel shaders and VBOs in UT2004.ini to add a bit more performance, but it's still marginal.

I'm rather disappointed. I'm not having nearly as great an experience as other Linux users, and certainly not as good as the Windows gamers who've benchmarked Unreal on this hardware. However, I did also catch the huge xorg-server 1.7 update as well, so maybe there have been some performance regressions since 1.6. It makes it a little hard to determine the areas that could use tweaking. I don't have anything special in my xorg.conf, just a default resolution. It's possible there's a setting I'm missing.

I'd like to try UT2004 on my desktop workstation, which has a RadeonHD 4550 card, but all reports indicate that even the latest git checkouts of the open-source drivers still don't work with Unreal. Apparently the game can't even launch, much less run at playable speeds. But as rapidly as the drivers are maturing, I'm hoping this'll be fixed in a month or so. Call me optimistic.

* * *

It looks like Abiword 2.8.0 was released today, so I wrote an ebuild and made it available in my devspace. I've been hand-writing these things for awhile. It took quite a bit of research to determine what went into the 2.7 betas, and now I'll have to do another overhaul of the 2.8 ebuild to account for the new plugin system. There's no longer a separate abiword-plugins package; they're all distributed in the base 2.8.0 archive. This means there will be a lot more tricky configure checks and USE flags, which sucks from a flexibility standpoint. Keeping the plugins in an external package was much simpler, so I'm a bit disappointed by this upstream decision.

Still, right now you can download and install Abiword 2.8.0 using my ebuild. While it needs a few cleanups, it will get you set up with a fully functioning basic Abiword install, though the only available plugin (as shown in the "Plugins" dialog) is .odt support.

This new version launches much quicker than 2.7.10, and it seems to have fixed all the rendering errors and even the crashes that happened with basic operations. Basically, you can click stuff now without worrying.

Cleaning up my ebuild is a long task, thanks to those darned plugins. Patches welcome, or I suppose you could always just wait and see what ends up in Bugzilla.

Original post from Planet Gentoo.

Remi Cardona: Xorg-server 1.7 in ~arch

Remi Cardona (remi) — Mon, 26 Oct 2009 21:50:42 +0000

It's out there now, available in ~arch. Like always, you'll need to rebuild your drivers, just look-up the command given by the server's ebuild (use eread if you've lost the output).

This release took a little longer to unmask not because of the server (it's a nice change). It's because a lot of headers were moved around from library packages to proto packages and vice versa. The ABI of X libraries has not changed, but I'm pretty sure there will be compile errors in some packages.

If that's the case, please file bugs in bugzilla.

Thanks for reading this public service announcement.

Edit: There will not be a package.keywords list for stable users. Xorg-server 1.7 is intended for ~arch users only, at this moment. And all bugs from stable users will be closed INVALID. We will start creating lists when we want to stabilize it.

Steve Dibb: new feeds

Steve Dibb (beandog) — Mon, 26 Oct 2009 17:46:23 +0000

I've been having a slew of issues running Apache on my Linode VPS, which I'm still trying to pin down, so in an attempt to offload some of the usage, I'm now going to use Feedburner to provide the RSS feed for Planet Larry.

I know I've played with Feedburner in the past, and kind of flip-flopped on whether to use it or not, but this time I'm sure I'm gonna stick with it. It's better for users, since they will always have a feed available (whether I have issues or not), and it's better for me since I can offload that part of the network traffic, which is actually quite a lot.

I've already updated the feeds and my apache config to do a permanent redirect, but if you want the feed URLs directly, here they are:

Planet Larry: http://feeds.feedburner.com/planet_larry
Larry the Universe: http://feeds.feedburner.com/universe_larry

Sorry for the inconvenience. It seems like everytime I post about Planet it's bad news or maintenance. Believe me when I say that it aggravates me far more than it does you.

Specifically, the issues I'm having is that Apache is sucking up all the available RAM, of which I only have 360 megs on my account. It's then rolling over to using all the swap space as well, which only slows things down even more. I've just started playing with tweaking the MPM configuration a bit, and I'm still trying to find a reasonable solution for my configuration.

In the past, the Linode had been seizing up occassionally, and I'd normally just reboot it and get on with my life. Recently, I installed monit (an awesome app), and pinpointed that the issue seems to always be with apache. Now, I'm just trying to narrow it down even more from there, but offloading the RSS feeds seems like a good step to take anyway ... I get gigabytes of traffic per month just on that, believe it or not.

I'm toying with the idea of setting up lighthttpd instead, but I really prefer apache, and would rather set it up to behave in a low memory environment instead. So, for any downtime in the near future, chances are it's just me tweaking something. At least now, thanks to monit, I have a much better idea of when something goes wrong.

Oh, one other tweak I've made is that the planet script itself is more robust as well. That thing used to run out of control, but I've made some changes that will ensure that if it runs away, at least it won't bring down the system. I also started playing around with the idea of writing my own feed parser to replace the Planet software completely, and it looks like it's going to be much simpler than I imagine. I haven't actually started down that path yet, since I have bigger projects to complete, but I'm actually enthusiastic that it'd be far, far simpler than I imagined.

Alex Legler: Gentoo at LinuxTag 2009 — Win free tickets!

Alex Legler (a3li) — Mon, 26 Oct 2009 15:35:38 +0000

From June 24 – June 27 there will be the LinuxTag in Berlin, one of the biggest Linux and Open Source events in Europe. Last year, almost 12.000 visitors came to the show, and for this year we hope that even more people will come.

Of course, you'll find Gentoo on the exhibitors list, special thanks to our main coordinator, Robert Buchholz (rbu) for organizing our presence.

Come visit the Gentoo folks in Hall 7.2a, Booth 101a (that is right next to the entrance!)

Bring your friends and maybe a little gift for the hard working Gentoo devs. ;) We are looking forward to interesting conversations!

To make the decision whether to come to LinuxTag even easier, here is a chance to…

Win free tickets! *

Here's the deal: Send an e-mail to a3li (at) gentoo (dot) org with the reason why you should get one of our 5 free tickets. Mabye you fixed a bug the other day or help our users a lot on IRC, be sure to include that reference.

We will pick the 5 submissions who we think have deserved it most. The competition ends Wednesday, June 17 at 2000 UTC.

The winners will be announced on Planet Gentoo and will receive an email with further instructions to claim the free tickets.

Nochmal auf Deutsch

Gentoo auf dem LinuxTag, vom 24. bis 27. Juni 2009. Ihr findet uns in Halle 7.2a, Stand 101a (gleich am Eingang).

Dazu verlosen wir 5 Freikarten, schreib eine E-Mail an a3li (at) gentoo (punkt) org und sag uns, warum gerade Du ein kostenloses Ticket verdient hast (vielleicht weil Du einen Bug gefixt hast oder anderen Benutzern häufig im IRC hilfst). Die fünf „fleißigsten” kriegen von uns Tickets. * Einsendeschluss ist Mittwoch, der 17. Juni 22.00 Uhr deutscher Zeit.

* There is no right of appeal. / Rechtsweg ausgeschlossen.

Planet Gentoo

Jorge Manuel B. S. Vicetto: Amarok-2.2.0/1 and >=binutils-2.20

Christian Faulhammer: Kernel testing request 2.6.31

Alex Alexander: Qt 4.6.0 rc1 – in portage – binary incompatibility warning!

Patrick Lauer: CPU Hotswapping and how to disable processors

Diego E. Pettenò: What distributions want

Sune Kloppenborg Jeppesen: Watching Freeview HD DVB-T MPEG-4 channels

Patrick Lauer: "Random crashes" with glibc 2.10 and 2.11

Patrick Lauer: Awesome portage options

Patrick Lauer: Configuring Portage

Diego E. Pettenò: Virtualisation WTF once again.

Patrick Lauer: HOWTO Radeon (opensource) + R700 + 3D / OpenGL

Gilles Dartiguelongue: Looking for a padawan

Doug Goldstein: NVIDIA legacy drivers update

Patrick Lauer: The OOM Killer (and how to make it less annoying)

Josh Saddler: November Xfce desktop

Alex Alexander: how-to: keep your kde 3.5 after it’s removed from gentoo’s tree, using the kde-sunset overlay

Josh Saddler: Tabu Audio Player

Patrick Lauer: The mistery of swappiness

Alex Alexander: gentoo-user-el mailing-list – for all the greeks out there :)

Diego E. Pettenò: Vacation time for me!

Doug Goldstein: MythTV 0.22 is out

Diego E. Pettenò: Plugins aren't always a good choice

Zhang Le: Update on building mips64el cross toolchain

Steve Dibb: packages website going offline for a while

Diego E. Pettenò: Debating future tinderbox work

Donnie Berkholz: links for 2009-11-06

Gentoo News: Gentoo KDE3 Deprecation Notice

Alex Alexander: KDE in Gentoo, an update

Diego E. Pettenò: Distributions *are* the strength of Linux

Steve Dibb: my blu-ray ripping trial run

Diego E. Pettenò: ELF should rather be on a diet

Steve Dibb: closed captioning on dvds (and ripping them)

Diego E. Pettenò: The PIE is not exactly a lie…

Diego E. Pettenò: Changes in the comment antispam

Damien Krotkine: Curses::Toolkit is on CPAN

Jeremy Olexa: Gentoo: About “optimizing”

Doug Goldstein: Improved VDPAU abstraction

Doug Goldstein: nvidia-drivers and xorg-server compatiblity

Jeremy Olexa: Using sshfs with rtorrent

Gilles Dartiguelongue: Gnome 2.28.1 full steam

Damien Krotkine: New Home

David Abbott: Podcast 65 Bashed In The Head

Remi Cardona: Response to a comment

Doug Goldstein: Subversion Branch Office Support aka commit-redir

Doug Goldstein: nvidia-drivers 190.x

Diego E. Pettenò: Blog posts are no replacement for documentation

Josh Saddler: Intel graphics and gaming, Abiword 2.8.0

Remi Cardona: Xorg-server 1.7 in ~arch

Steve Dibb: new feeds

Alex Legler: Gentoo at LinuxTag 2009 — Win free tickets!

Win free tickets! *

Nochmal auf Deutsch

Diego E. Pettenò: Distributions are the strength of Linux