#if defined(__LITTLE_ENDIAN) && defined(__BIG_ENDIAN)
# error Fix asm/byteorder.h to define one endianness
#endif

A new sub-series of For A Parallel World, to give some development pointers on how to make software make better use of parallel computing resources available with modern multicore systems. Hope some people can find it interesting!

While I was looking for a (yet unfound) scripting language that could allow me to easily write conversion scripts with FFmpeg that execute in parallel, I’ve decided to finally take a look to OpenMP, which is something I wanted to do for quite a while. Thanks to some Intel documentation I tried it out on the old benchmark I used to compare glibc, glib and libavcodec/FFmpeg for what concerned byte swapping.

The code of the byte swapping routine was adapted to this (note the fact that the index is ssize_t, which means there is opening for breakage when len is bigger than SSIZE_T_MAX):

void test_bswap_file(const uint16_t *input, uint16_t *output, size_t len) {
  ssize_t i;

#ifdef _OPENMP
#pragma omp parallel for
#endif
  for (i = 0; i < len/2; i++) {
    output[i] = GUINT16_FROM_BE(input[i]);
  }
}

and I compared the execution time, through the rdtsc instruction, between the standard code and the OpenMP-powered one, with GCC 4.3. The result actually surprised me. I expected I would have needed to make this much more complex, like breaking it into multiple chunks, to be able to make it fast. Instead it actually worked nice running one cycle on each:

The results are the average of 100 runs for each size, the file is on disk accessed through VFS, it’s not urandom; the values are not important since they are the output of rdtsc() so they only work for comparison.

Now of course this is a very lame way to use OpenMP, there are much better more complex ways to make use of it but all in all, I see quite a few interesting patterns arising. One very interesting note is that I can make use of this in quite a few of the audio output conversion functions in xine-lib, if I was able to resume dedicating time to work on that. Unfortunately lately I’ve been quite limited in time, especially since I don’t have a stable job.

But anyway, it’s interesting to know; if you’re a developer interested in making your code faster on modern multicore systems, consider taking some time to play with OpenMP. With some luck I’m going to write more about it in the future on this blog.

After my post about AppleTV conversion with FFmpeg I’ve been working on trying to streamline my conversion procedure even further. The low-quality of the conversion is rarely an issue for me since what I’m uploading in there is usually something I watch before I can get to the DVDs. I have sometimes to add borders or subtitles would disappear over the edge of the TV, but not always since the content might not be subtitled.

Unfortunately it seems like GNU make has issues where paths with spaces are involved, and there are some other things that are clumsy and require me to write the same rule twice to get it to properly work. At the end it doesn’t seem like GNU make is the right tool for the job, so I gone looking for something different.

The rake option was discarded right away since rake does not parallel-execute, so I went on looking for something. I just needed a make-workalike, rather than a build system, or even a scripting language with proper support for parallelising, something like bash’s for construct but having a number of concurrent jobs applicable to it.

Looking for that in Google I came through an article about Make alternatives on freshmeat. It looked promising at the start but then committed the mistake of confusing make itself and higher level build systems, and went on for most of the time to speak about alternative build system. It still had a few names, among which I took out cook that at first looked interesting.

The syntax seems to be similar enough to make to be easy to grasp, and still allows much more flexibility than the standard make. So what’s the catch? Well, the catch begins with it using yet another strange SCM (like Pidgin’s monotone wasn’t annoying enough), but it doesn’t stop here.

Out of habit I check the ebuilds when I want to try new software, it helps to know whether the original code is so messy that we have to patch the hell out of it; in addition to this for buildsystems, having complex ebuilds means that the build system architects themselves don’t have very clear ideas on what it’s needed. Seems like this was a good idea:

src_compile() {
        econf || die "./configure failed"
        # doesn't seem to like parallel
        emake -j1 || die
}

src_install() {
        # we'll hijack the RPM_BUILD_ROOT variable which is intended for a
        # similiar purpose anyway
        make RPM_BUILD_ROOT="${D}" install || die
}

So a make-replacement that on its homepage declares to be able to build in parallel… has a build system that does not build in parallel? Indeed the makefile is an absolute mess; while it’s true that you don’t need to know one tool to write a replacement for it (that is not a drop-in replacement), it still would help. I could understand the mistake with multiple output rules since the cook syntax diverges from make’s in that regard; I started understanding much less the mistakes about temporary file creation, but I was really put off by the fact that it does not even know how to use the -o switch at the compiler to get it to output to a filename that is different from basename.o.

Let me say one thing: before you decide to write a tool that is better than $something, try at least to know $something well enough so that you don’t reinvent the wheel, squared.

Before I resume working on PAM (I need to implement a change to pam_lastlog to fix a pernicious bug), I wanted to just write a quick entry for the paranoid of you who still use PAM for system login.

Since, as you most likely already know, MD5 is once again considered insecure, one obvious concern would be the fact that passwords saved in MD5 on a system are not secure either. For this reason if you’re using Linux-PAM, you can make use of the SHA512 hashing of system password keys, which I already wrote about.

Remember that to use that you have to make sure your Linux-PAM (sys-libs/pam) is built against a recent enough version of glibc. Unfortunately the version of pambase with this feature hasn’t hit stable yet, the bug above is blocking it, and I’m going to have to hack at pam_lastlog to fix that.

What I didn’t write last time, is that you can easily spot if your system is using md5 passwords by using this simple command from root:

# fgrep '$1$' /etc/shadow

Of course one has to access your /etc/shadow file to breach your passwords, so your system has to have been compromised before, but it’s still not nice if they can find out what your basic passwords are.

Moving on.

After a longish time, here for you a new chapter of my widely read series For A Parallel World, improving buildsystems to reduce build time on modern multiprocessor, multicore systems.

This time, rather than the usual build failures, I’m going to speak of a parallel install failure. Even though one can think of install as a task that rarely can fall into problems like race conditions and the such, and even though it’s probably the part that gets less boost when using parallel make on a multicore system (since it’s usually I/O bound rather than CPU bound), it’s actually one very fragile part of many packages.

One of the common failures is due to old install-sh script used to simulate the install command on systems too old to have a POSIX-compatible one, and which is also used to create directories recursively if mkdir -p is missing. For a series of reason, this hits pretty often on FreeBSD, but this is beside the point. This can be easily solved by replacing the old faulty script with an updated copy out of automake or libtool, which does not have problems at all.

A few times, the problem is instead due to a broken Makefile.am. Let’s take a practical example from some software I fixed recently after being called in action by nixnut: gramps . Please note that if you look at the bug now you’re going to spoil the post, since it contains the solution straight away, while I’m going to explain it step by step.

Let’s start from the reported build log:

test -z "/usr/share/gramps/docgen" || /bin/mkdir -p
"/var/tmp/portage/app-misc/gramps-3.0.3/image//usr/share/gramps/docgen"
 /usr/bin/install -c -m 644 'gtkprintpreview.glade'
'/var/tmp/portage/app-misc/gramps-3.0.3/image//usr/share/gramps/docgen/gtkprintpreview.glade'
 /usr/bin/install -c -m 644 'gtkprintpreview.glade'
'/var/tmp/portage/app-misc/gramps-3.0.3/image//usr/share/gramps/docgen/gtkprintpreview.glade'
/usr/bin/install: cannot create regular file
`/var/tmp/portage/app-misc/gramps-3.0.3/image//usr/share/gramps/docgen/gtkprintpreview.glade':
File exists
make[3]: *** [install-docgenDATA] Error 1
make[3]: *** Waiting for unfinished jobs....

As usual, the first thing we’re looking for when there is a parallel build (or install) failure are repeated commands. As I’ve shown in Case Study n. 2, when the same command is repeated multiple times it’s often due to mistakes in the Makefiles, thus before thinking of a problem with the dependencies, I check for that. It’s way more common especially on automake-based build systems.

So indeed we can see there are two calls to the install command for the file gtkprintpreview.glade (this also shows us that it’s not a problem of old and faulty install-sh script since the call is directly to the system command). Contrary to what happens when it’s a build rule that is wrongly expressed in the makefile, the double-call during install phase is usually present both using parallel jobs and not. The difference is that when the two calls happen sequentially, the second just overwrites the results of the first; wastes time but it’s successful. On the other hand when parallel jobs are used, the two calls are often enough happening at the same time, and thus we have a race condition.

Okay so next step as usual is to look at the incriminated Makefile.am:

[snip]
docgen_DATA = \
        gtkprintpreview.glade

dist_docgen_DATA = $(docgen_DATA)
[snip]

Here we’re at the core of the problem. The gtkprintpreview.glade file is part of the sources, and it has to be installed as part of the docgen class of files (thus in $docgendir). But the data installed in that path is listed twice, once in the docgen_DATA variable and one in dist_docgen_DATA, causing the file to be installed twice on two independent targets. Since the two targets are independent, when using parallel jobs they both will run at the same time the same command.

Let me try to explain what the mistake has been. By default the sources are packaged up in the final tarball, if they are not generated by rules from the make process; sometimes you wish files that are built by make to still be distributed, and thus you either have to use EXTRA_DIST or prefix dist_ to the class of the installed files, to explicit that the files have to be distributed. Unfortunately the gramps developers didn’t know automake well enough, and thought that dist_docgen_DATA worked quite a lot like EXTRA_DIST (maybe it actually used EXTRA_DIST in the past, for what I know), and thus duplicated the variable content.

The solution? Just replace the use of docgen_DATA with dist_docgen_DATA and remove the second definition, the problem is solved at the source.

If you’ve been following my blog for a while you probably remember how much I fought with VirtualBox once it was released to get it to work, so that I could use OpenSolaris. Nowadays, even with some quirks, VirtualBox Open Source Edition is fairly usable, and I’m using it not only for OpenSolaris but also for a Fedora system (which I use for comparing issues with Gentoo), a Windows install (that I use for my job), and a Slackware install that I use for kernel hacking.

Obviously, the problem is that the free version of VirtualBox come with some disadvantages, like not being able to forward USB devices, having limited type of hardware to virtualise and so on. This is not much of a problem for my use, but of course it would have been nicer if they just open sourced the whole lot. I guess the most obnoxious problem with VirtualBox (OSE at least, not sure about the proprietary version) is the inability to use a true block device as virtual disk, but rather having to deal with the custom image format that is really slow at times, and needs to pass through the VFS.

For these reasons Luca suggested me many times to try out kvm instead, but I have to say one nice thing of VirtualBox is that it has a quite easy to use interface which allows me to set up new virtual machines in just a few clicks. And since nowadays it also supports VT-x and similar, it’s not so bad at all.

But anyway, I wanted to try kvm, and tonight I finally decided to install it, together with the virt-manager frontend although there are lots of hopes for this, it’s not yet good enough, and it really isn’t usable for me at all. I guess I might actually get to hack at it, but maybe this is a bit too soon yet.

Continue reading on my blog for the reasoning, if you’re interested.

One thing I really dislike of the newer versions of VirtualBox is that it went the VMware route and decided to create its own kernel module to handle networking. They said it’s for performance reasons but I’d expect the main reason is that it would allow them to have a single interface between Linux, Windows, Solaris and so on. The TUN/TAP interface is Linux-specific so supporting that together with whatever they have been doing on the other operating systems is likely a cost for them. Although I can understand their reasoning, it’s not that I like it at all. More kernel code means more code that can crash your system, especially when not using the Open Source Edition.

Interestingly enough, the RedHat’s Virtual Machine Manager is instead doing its best to avoid creating new network management software, and uses very common pieces of software: dnsmasq as DHCP server, the Linux kernel bridging support, and so on. This is very good, but it also poses a few problems: first of all, my system already runs a DHCP server, why do I have to install another? But it’s not just that; instead of creating an userspace networking bridging software, like VirtualBox does, it tries to use what the kernel provides already, in the form of the bridging support and iptables to forward requests and create NAT zones. This is all fine and dandy, as it reduces the feature duplication in software, but obviously requires more options to be enabled in kernels that might not otherwise have it enabled at all.

As it happens, my system does have bridging support installed, but not iptables nor masquerade targets and similar. Up to now I never used them so it had no reason to be there. I also sincerely can’t understand why it does need it if I don’t want a NAT, but it doesn’t seem to allow me to set up the network myself. Which would include me being able to just tell it to create a new interface and add it to a bridge I manage myself, leaving to me all the details like dhcp, and thus not requiring iptables at all.

Also, even though there is a way to configure an LVM-based storage pool, it does not seem to allow me to choose directly one of the volumes of that pool when it asks me what to use as a virtual disk. Which is kinda silly, to me.

But these are just minor annoyances, there are much bigger problems: if the modules for kvm (kvm and kvm-amd in my system) are not loaded when virt-manager is loaded, it obviously lack a way to choose KVM as hypervisor. This is nice, but it also fails to see that there is no qemu in the system, and tries to use a default path, that is not only specific to RedHat, but also very not existing here (/usr/lib64/xen/bin/qemu-dm, on x86-64, just checking the uname!), returning an error that the particular command couldn’t be found. At least it should probably have said that it couldn’t find qemu, rather than the particular path. It would have also been nice to still allow choosing kvm but then telling that the device was missing (and suggested loading the modules; VirtualBox does something like that already).

To that you have to add that I haven’t been able to finish the wizard to create a new virtual machine yet. This because it does not check the permission to create the virtual machine before proposing you to create one, so it let you spend time to fill in the settings in the wizard and then fails telling you don’t have permission to access the read/write socket. Which by default is accessible only by root.

Even though it’s obvious by the 0-prefix that the software is still not production-level, there are also a few notes on the way the interfaces are designed. While it’s a good idea to use Python to write the interface, since it allows a much faster test of the code and so on, and speed is not a critical issue here, as the work is carried out by C code in the background, every error is reported with a Python traceback, one of the most obnoxious things to do for users. In particular for the permission problem I just listed, the message is a generic error message: “Unable to complete install: ‘unable to connect to ’/var/run/libvirt/libvirt-sock’: Permission denied’”; what is the user supposed to know here? Of course virtualisation is not something the user at the first tries with Linux is going to use, but still, even a power user might be scared by errors that appear this way and have attached a traceback (that most users would probably rationally link with “the program crashed” like bug-buddy messages, rather than “something wasn’t as the software expected”, which is what happened.

On a very puny level, instead, I have to notice that just pressing Enter to proceed to the next page on the wizard fails, and using ESC to close the windows too. Which is very unlike any other commonly-used software.

So to cut the post short now, what is the outcome of my test? Well, I didn’t get to test the actual running performance of KVM against VirtualBox, so I cannot say much about that technology. I can say that there is a long road ahead for the Virtual Machine Manager software to become a frontend half as good as VirtualBox’s or VMware’s. This does not mean that the software was badly written, at all. The software by design is not bad at all; of course it’s very focused on RedHat and less on other distributions, which explains lots of the problems I’ve noticed, but in general I think it’s going the right way. A more advanced setup for advanced users would probably be welcome by me, as well as an ISO images management like the one VirtualBox has (even better if you can assign an ISO image to be of a specific operating system, so that when I choose “Linux”, “Fedora” it would list me the various ISO for Fedora 8, 9, 10 and then have a “Other” button to take a different ISO to install, if desired.

I’ll keep waiting to see how it evolves.

I’ve been told quite a few times that my posts tend to be too long, and boring for most basic users, so for a time I’ll try to use the “extended content” support from Typo, and see how people react. What this means is that the blog post is just summarised on feeds, and on aggregators like Planet, while the complete text can be read by accessing the article directly on my blog.

When I started my work reporting bundled libraries almost an year ago, my idea had a lot to do with sharing code and just to the side to do with the security issues related to bundled libraries. I had of course first hand experience with the problem, since xine-lib has (and still in part had) bundled a lot of libraries. When I took over maintainership of it in Gentoo, it was largely breaching policy, and the number of issues I had with that was huge. With time, and coordination with upstream (to the point of me becoming upstream), the issues were addressed, and nowadays most of xine-lib bundled libraries are ignored in favour of the system copies (where possible; some were largely modified to the point of not being usable, but that’s still something we’re fighting with). Nowadays, the 1.2 branch of xine-lib already doesn’t have a FFmpeg copy at all, always using the system copy (or an eventual static copy built properly).

But nowadays I started to see that what is obvious to me about the problems with bundled copies of libraries is not obvious to all developers, and even less obvious to “power users” who proxy-maintain ebuilds and just want them to work for them, rather than complying with Gentoo policies and standards. Which is why I think that sunrise and other overlays should always be scrutinised carefully before being added to a system.

At any rate, for this reason I’m going to explain in this post why you should not use bundled internal copies of libraries for packages added to Gentoo, and why in particular these packages should not be deemed stable at all.

The first issue to discuss is why do upstreams bundle libraries, since knowing the reasoning behind that is often helpful to identify whether it makes sense at all to keep them or not. The first most obvious answer is: to reduce dependencies. For a long time this was the major reason behind xine-lib usage of internally bundled libraries. As it turns out, with time this reason became moot: distributions started packaging xine-lib directly reducing the number of users wishing to build it from sources; even those wanting to build xine-lib from sources would find all the needed libraries in the distributions, most of the times. When this is the sole reason for libraries building, upstream should be very well open to add a configure option (or anything similar to that) to use the system copy, optionally or by default, with fallback to the bundled copy.

A second reason might be that the library is being unstable when it comes to API; this is probably the first reason why FFmpeg is often bundled in software rather than using the copy on the system; while this is a concern that makes more sense then the one before, it’s still mostly a moot point since it really just requires to fix the issue at the source: get the original project to maintain API compatibility, to provide an API compatibility layer, or to finalise its API. Even when it cannot be helped, because the API is in flux, maintained software fears not the API break; it might be a bit of an hassle but in general it’s feasible to keep the use and the library in sync.

Thirdly, more worrisome, is when the library is modified, slightly or heavily, after bundling; in this case using the system copy might be quite a burden because it will lack the specific changes as made by the project. In this case there is a lot of work involved, sometimes more work that it can be taken care by distributions, and requires coordination of the project’s upstream together with the higher level upstream. This is what happened with xine-lib and FFmpeg: the copy in the xine-lib sources was heavily modified to suit both the build system and the interface requirements of xine, which made it also very difficult to update the internal snapshot. All the interface changes needed have then been pushed upstream to FFmpeg, and the buildsystem changes were made moot by using the default buildsystem (with needed changes pushed upstream) embedded in autotools; and then FFmpeg was entirely removed from xine-lib’s sources.

Now, on the other hand, the disadvantages of using bundled libraries are probably worse: code duplication means that there is more data to process (both at build time to compile and at load time to store in memory), there is more space used by the binaries, and there are duplicated bugs that need to be fixed twice. A lot of time in xine-lib the problems with decoding something with FFmpeg were solved by just using a newer FFmpeg; why keeping one then?

The most important issue though is about security: when a vulnerability is found in a library like zlib, fixing the library alone is not enough: while that fixes the majority of the software in a system, it’s not going to fix those who bundle it, both closed-source and open source. For instance, take dzip it uses an ancient internal version of zlib; if somebody knows the format well enough, it’s far from impossible to craft a dzip file that contains a deflated stream that can executed malicious code.

For this latter issue alone, I’d say that any software bundling source code is not good enough to go stable on its own. Of course sometimes one has to bend the rules because of past mistakes, for instance even though Ruby bundles stuff, we cannot stop newer versions to go stable; this problem is not a regression. But should stop other broken software from entering portage or at least the stable tree.

But it’s not just security, subtle bugs might actually be quite a problem. For instance, you might remember all Java applications failing when libX11 was built with XCB support some time ago. The problem was due to some stricter checks in libxcb compared to what libX11 have been checking before, but the source of the problem was Xinerama. The problem with that was that Sun bundled an internal copies of libXinerama sources in the JRE sources, and even though libXinerama was since then fixed regarding that particular issue (the crash with XCB), it was never updated in the JRE before the issue became a nuisance for users.

A very similar issue, also involving X11 (just by chance, it’s not that all the issues involve X11) is this particular bug in Xorg that is triggered when launching SDL-based applications, because libSDL bundles ancient versions of X11 libraries.

As I said earlier, unbundling is rarely easy; there are subtle issues to be checked out, for instance one has to check if there are changes at all beside eventual build-system related things (for instance to avoid using a full-fledged ./configure), but altogether it’s usually not tremendously impossible. Of course one has to stop thinking “Oh my, what if a library changes and the software breaks?”, otherwise the task gets impossible. Software changes, software bitrots. It’s not by bundling internal copies of libraries that you can stop that. When the compiler gets upgraded, you’re going to have your software break, and you should fix your software; if the C library cleans up the includes, your software might not compile or might misbehave, deal with it. Sometimes the bundled libraries implement protocols and formats that need to work together with some other piece of software; if that changes, the bundled libraries are just going to break further.

Your software is rarely special enough that you can be exempted from following the rules. Even OpenOffice is using lots of system libraries nowadays!

Bundling and modifying libraries is just like forking a project, and forking might not always be the best approach sometimes with dead upstream for a project, forking is your only hope; but even in those cases there are nice ways to bundle libraries. If you look at the way nmap bundles libdnet, you can see that they not only document all the changes they made to the library, but also provides splitted down and commented patches for the various changes, making it possible to adapt it to their need.

For proprietary software packages, of course, the matter is different, since you cannot usually unbundle the libraries yourself; but it’s a good idea to ask upstream nicely if they can use system copies instead of internal ones. Mind you, some might be happy to fix their packages not to be vulnerable any longer. Although I guess lots of them might actually prefer to keep them as they are since it’s a cost to them. One more reason not to trust them, to me.

So bottom line, if you’re working on an ebuild for a new piece of software to submit for Portage addition, please look well to see if the software is bundling libraries, and if it is, don’t let it enter portage that way. And if you’re a developer who wants to push some ebuild to the tree, also remember to ensure that it complies with our policies and doesn’t bundle in libraries.

I’ve already written about the common mistake of using AC_CANONICAL_TARGET in software that is not intended to be used as compiler. Since I’m now using my tinderbox extensively, I’ve thought it might have been a good idea to try checking how many packages actually do call that, that shouldn’t.

The test is really a quick and dirty bashrc snippet:

post_src_unpack() {
    find "${S}" -name configure.ac -o -name configure.in | while read acfile; do
        acdir=$(dirname "$acfile")
        pushd "$acdir"
        autoconf --trace AC_CANONICAL_TARGET 2>/dev/null > "${T}"/flameeyes-canonical-target.log
        if [[ -s "${T}"/flameeyes-canonical-target.log ]]; then
            ewarn "Flameeyes notice: AC_CANONICAL_TARGET used"
            cat "${T}"/flameeyes-canonical-target.log
        fi
        popd
    done
}

This provides me with enough information to inspect the software, and eventually provide patches to correct the mistake. As I said this is a very common mistake, and I’ve fixed quite a few packages for this. Not only it wastes time to identify the target system, but it also provides a totally misleading --target option to the configure script that confuses users and automatic systems alike; if we were to write a software to generate ebuilds out of the source tarball of a software with some basic common options, it would probably be confused a lot by the presence of such an option.

Since the whole build, host and target situation is often confusing, I’d like to try explaining it with some images, I think that might be a good way to show users and developers alike how the three machine definitions interact between each other. Since this is going to be a long post, in term of size, rather than content, because of the images, the extended explanation won’t be present in the feed.

Continue reading on my site for that.

To try explaining this in a very visual way, let’s say we have only three systems, a PowerBook laptop, a standard x86 computer, and a build service using x86-64 servers. The choice of PoweBook as smallest device has been conditioned by the fact it was the only decent image I could find on OpenClipart for a system that would have been easily seen as having a different architecture than the other two. I would have liked an ARM board, but it was wishing too much.

The first obvious case is having a native compiler, no cross-compiling involved at all:

In this case, both gcc’s configure script, the powerpc-linux-gnu-gcc compiler and the hellow program are executed on the same system: the laptop. This is the standard case you have on a Gentoo system when building stuff out of most ebuilds. In this case host, build and target machines are all one the same: powerpc-linux-gnu.

Then there is a very common case for embedded developers, cross-compilers:

In this case gcc’s configure (and thus build) is executed on a PC system, which also will run the powerpc-linux-gnu-gcc compiler, but the hellow program is still executed on the laptop. Host and build machines are i686-pc-linux-gnu while target is powerpc-linux-gnu.

The next scenarios is uncommon for standalone developers but is somewhat with binary distributions for smaller systems:

In this case there is a build service that starts up the build for the compiler, that will then be executed by the laptop directly. In this case the build machine is x86_64-pc-linux-gnu while both host and target are powerpc-linux-gnu.

The final scenario involves all three systems at once and shows exactly the difference between the three machine definitions:

In this case the build service prepares a cross-compiler executed on a PC that will build software to run on the laptop. The build machine is x86_64-pc-linux-gnu, the host machine is i686-pc-linux-gnu and the target is powerpc-linux-gnu.

Now, this works pretty well and sleek for compilers, but what about other software? In most cases you got just two systems involved at most, one that will run the software and one that will build it, so there is no need for a target definition, it’ll all be completed between build and host. And this is why you should not be calling AC_CANONICAL_TARGET unless you can figure out a far-fetched scenario where you can involve three computers with three different architectures, like in the last scenario.

Today, even though it’s the new year’s eve, I’ve spent it working just like any other day, looking through the analysis log for my linking collisions script, to find some more crappy software in need of fixes. As it turns out, I found quite a bit of software, but I also confirmed to myself I have crappy database skills.

The original output of the script, already taking quite a long time to process, didn’t sort the symbols by name, but just by count, so to show the symbols with most collisions first and the ones that related to one or two files later. It also didn’t sort the name of the objects where the symbols could be find, which caused quite an issue as from time to time the list changed sorting so the list of elements wasn’t easy to compare between symbols.

Yesterday I added sorting to both fields so that I could have a more pleasant og to read, but it caused the script to slow down tremendously. At which point I noticed that maybe, just maybe, PostgreSQL didn’t optimise my tables, even though I had created views, in the hope of it being smart enough to use them as optimisation options. So I created two indexes, one for the name of the objects and one for the name of the symbols, with the default handler (btree).

The harvesting process now slowed down of a good 50%. Instead of taking less than 40 minutes, it took about an hour, but then when I launched the analysis script, it generated the whole 30MB log file in a matter of minuts rather than requiring me hours, I never have been able to let the analysis script complete its work before, and now it did it in minutes.

I have no problem to say that my database skills suck, which is probably why I’m much more of a system developer than a webapp developer.

Now at least i won’t have many more doubts about adding a way to automatically expand “multimplementations”: with the speed it has now I can well get it to merge in the data from the third table without many issues. But still, seeing how much my SQL skills are pointless, I’d like to ask some help on how to deal with this.

Basically, I have a table with paths, each of which refers to a particular object, which I call “multimplementation” (and groups together all the symbols related to a particular library ignoring things like ABI versioning and different sub-versions). For each of the multimplementation I have to get a descriptive name to report to users. When there is just one path linked to that object, that path should be used; when there are two paths, the name of the object, plus the two paths should be used; for more than two paths, the object name and the path of the first object should be used, with ellipses to indicate that there are more.

If you want to see the actual schema, you can find it on ruby-elf’s repository in the tools directory.

There are more changes to the database that I should do to make it much more feasible to connect the paths (and thus the objects) to the package names, but at least now with the speed it took it seems to be feasible to run these check on a more stable basis on the tinderbox. If only I could find an easy way to have incremental harvesting, I might as well be able to run it on my actual system too.

There is one thing I noticed working on my linking collision script. While most of the software properly creates subdirectories to put plugins in, so that they don’t clash with others and it does not pollute the LDPATH space, there are quite a few packages that don’t do that at all and install their plugins straight into the libdir.

Not only that, some packages install static archive versions of their own plugins, with no good reason since they are never linked in statically, but always dlopen’d.

Please don’t pollute LDPATH, if you can, make sure the plugins are installed in “pkglibdir” (that is /usr/lib/packagename) and make sure that they only install the shared object file, and eventually the libtool archive if the software uses libltdl to load them. The static archive is almost always unneeded and just a waste of time.

Also please remember that if you install core libraries in a path outside of the standard libdir (which is very good if the libraries are not to be linked against!), you should probably make sure that there is a proper runpath in the executables. What runpath does is to tell the linker to look for libraries in a path that is otherwise not accessible through the standard configuration files (/etc/ld.so.conf). A common mistake here is to install an env file that makes LDPATH (or even worse LD_LIBRARY_PATH) to the directory where the core internal libraries are installed.

While this works, it does not make much difference than having it in the standard library path: both the runtime linker and the link editor will use the path from the configuration file anyway, so the library is not going to be hidden like you’d want it to for a private library. On the other hand, if just the executable provides their own runpath, then the two linkers will ignore the libraries altogether.

So please, be careful with what you push in the library path, okay?

I’ve already written about some of the differences between my and Patrick’s tinderboxes, one of which is that my tinderbox does not only install one package, but tries to install all of them at once. This is a necessity for me to have enough material to work on with my library collisions script, and still have so many side effects that makes it funny to work with.

The first problem is that sometimes packages don’t get merged together because of file collisions, which most of the time are caused by packages that install commands with name too generic, and some other times because they regenerate files that should not be present in the final image (like iconv’s charset.alias that gets generated on Gentoo/FreeBSD systems with no good reason at all).

The second problem derives from the way the first problem is handled. When two packages install a file with the same name, rather than renaming one of them or both, it’s customary to actually “fix” the problem by adding blockers to each of the two packages so that they cannot get installed together. While it’s certainly better to have it expressed that way rather than having the merge to fail after the compilation and install phases, it’s not really a solution since it still disallows having the two packages on the same system. While this is acceptable for packages like the different GhostScript implementations that apply for the same task, this is not much of a solution when the packages are entirely independent one from the other and have very different tasks.

I also have found one particular package (pnet) which had a very funky solution to the collision between that and boehm-gc, considering that it was installing a private copy of that. Obviously this was not the proper fix by a mile’s look.

If you have two packages that block each other you have a few different ways to deal with that; if they provide the same function, you might as well install them with a prefix and then write an eselect module to choose between one or the other (which is something that ghostscript could very well be doing); if they only install executables with generic name, they might be changed to prefix the command name with the package name. But sometimes these commands are not to be used by the users at all, and are rather internal commands used by the scripts for processing; in those cases, it would be a nice idea to make those get into /usr/libexec/$PN/ so that they are taken out of the users’ path, and won’t collide one with the other.

While dealing with packages that install colliding files is not so easy, there is need for developers to deal with them in a less “works for me” way, and think more of the general picture, as it is, there are enough packages in the tree that blocks each other with no real good reason, and this is upsetting.

For a while I’ve been using GMane for almost all the mailing list I’m following. This made it much easier to deal with it because it mean that I didn’t have to download a local copy of all the messages and it also allowed me for a much cleaner access to archives. Unfortunately this has a downside, as it requires me to use an NNTP client, which is something that hasn’t been much cool to do lately.

In the last few few months I’ve used gnus as my client, but the problem with that is that it still needs a separate software from Evolution (which is my mail client0, and it had the nasty issue of not saving the read messages when emacs closed unexpectedly, say when X died, and because of a bug, emacs daemon died with it.

Now of course I could use mailing lists like most of the other people in the world do, by receiving them on my mail account, but I’d rather not fill my GMail “All Mail” folder with all the messages coming from mailing list, especially for when I have to access that data from an UMTS connection where I pay the traffic.

What other solutions are there for me? I’m considering the idea of doing what I did when there was a limit of 20MB on an email account from my provider, but no limit on the number of accounts, having one account per mailing list. Now the limit is 1GB but I haven’t been using those accounts in quite a long time. Even if they are available in IMAP when I’m on their connection they are only available through webmail from outside; not like that’s too much of an issue for me, since I need mailing list mostly when I’m not around hospitals or stuff like that, so I could just use that.

For this to work, though, I need a few tools working on IMAP, for instance I’d need a script that could expunge old archives, when the threads haven’t been updated in the last few months; I’ll also need a script that would automatically filter the mailing list as they arrive in Inbox (waiting with the IDLE command), since my provider does not allow me to filter the messages server-side. And such a script will have to run on Yamato since it has to be on their network.

Then there is the problem that Evolution saves its cache in my home directory, which is under RAID6; there is no need for the cache to be on my home, when XDG_CACHE_DIR is pointing to a private subdirectory in /var/cache. This includes the 200MB of SQLite database that is currently using. Does anybody know if there is a way to get Evolution to respect the XDG_CACHE_DIR variable?

These holidays really sucked, for a long series of reasons, and in general, I’m not feeling well neither emotionally nor physically. But they offered me the time to think about what I want to do. I’ve been working on my collision detection script lately and I’m now confident I can make it work as a proper tool to identify issues, and I’d love to work on fixing those issues with upstream, but the problem is I lack the time to do that.

Even if I do improve a project a day, it’s never going to be enough, because in a few months I’m going to need to do it again because code would have rot and stuff like that. I need help for this. One thing I’m going to do is working on a personal archive of autoconf macros I can use on different projects; the attributes.m4 file that comes with xine, lscube and Lennart’s projects is already a personal archive of macros under some aspects, the problem is that its history is shared among different repositories which is very nasty. I’ve avoided up to now to create a repository for that, splitting it up in different macro files (since it’s far from being just attributes checking any longer), but I think i should look for a solution for that problem rather than continuing to procrastinate that.

Today I stopped procrastinating on getting rid of JFS for my root filesystem: since kernel 2.6.28 was released, I’m now starting the long awaited conversion of my partitions to ext4, starting from the repositories (and here git wins against Mercurial big time: once the git repositories are repacked, copying them over is very very quick), while copying over openjdk, icedtea and xine repositories that use Mercurial take so much longer.

Talking about xine, I’m going to do some more work on that in the next days, mostly code cleanup if I can, but I’m also planning on setting up a Transifex instance on this server for xine (and my own projects); hopefully it’ll make it simpler to provide translated versions of xine-lib, xine-ui and gxine. As well as of my own tools that need to be translated one way or another.

There are so many things I’d have to do that I haven’t been able to in months, reading is one of those, but I’m going to preserve that for when I’m going to the hospital next month for check ups; I’m not going to bring my laptop with me this time, nor any handheld console. I’ll be around on the cellphone a bit maybe, but that’ll be it. For the rest of the time I’ll be reading and listening to music (I’m not going to leave the iPod at home, knowing hospitals it will come handy). Actually, since I just have to have a CT scan, a chest X-Ray and an MRCP (MRI), I don’t strictly need to stay in the hospital; but not having a driving license does not help; although I guess even if I had one, I’d better not be driving after they have tests on me.

I’m going to spend the new year’s eve alone at home, maybe with my mom and my sister with her husband and my nephew. On the whole, it’s not going to be holiday either, so like it happened on Christmas, I’m going to spend most of the day working on some analysis or similar. I’ve seen that some of the issues I’ve brought up lately have started being taken care of, which is very good.

I know this post sounds pretty incoherent, I guess I’m incoherent myself at the moment. Anyway if you wish to help out with anything at all, feel free to drop a line.

So it’s Christmas day, and I’m skimming through the 32+ MB of logs generated by my elven script and I found some nice nuggets:

Symbol getopt_long@@ (32-bit UNIX System V ABI Intel 80386) present 35 times
  /usr/bin/graph
  /usr/bin/ebzipinfo
  /usr/bin/spline
  /usr/bin/double
  /usr/bin/uupick
  /usr/bin/autotrace
  /usr/bin/uustat
  /usr/sbin/ndtpd
  /usr/bin/ode
  /usr/sbin/ndtpcheck
  /usr/bin/ebrefile
  /usr/bin/uucp
  /usr/sbin/ndtpcontrol
  /usr/bin/uulog
  /usr/sbin/uuxqt
  /usr/bin/ebzip
  /usr/bin/faad
  /usr/bin/lha
  /usr/lib/libsox.so.1.0.0
  /usr/sbin/uucico
  /usr/bin/ebinfo
  /usr/sbin/uuconv
  /usr/bin/plot
  /usr/bin/tek2plot
  /usr/lib/libc.so.5
  /usr/bin/uuname
  /usr/bin/uux
  /usr/bin/ebstopcode
  /usr/bin/stklos
  /usr/bin/cu
  /usr/bin/plotfont
  /usr/bin/ebfont
  /usr/bin/ebunzip
  /usr/bin/pic2plot
  /usr/sbin/uuchk
Symbol strncasecmp@@ (32-bit UNIX System V ABI Intel 80386) present 5 times
  /usr/bin/xpilots
  /usr/bin/gargoyle-agility
  /usr/lib/libc.so.5
  /usr/bin/xedit
  /usr/bin/xpilot
Symbol strnlen@@ (32-bit UNIX System V ABI Intel 80386) present 5 times
  /usr/bin/tarsync
  /usr/lib/libCw.so.1.0.0
  /usr/lib/libmba.so.0.9.1
  /usr/lib/python2.5/site-packages/numarray/_chararray.so
  /usr/bin/linksys-tftp
Symbol stricmp@@ (32-bit UNIX System V ABI Intel 80386) present 5 times
  /usr/bin/qemacs
  /usr/lib/libraidutil.so.0.0.0
  /usr/lib/openbabel/2.2.0/inchiformat.so
  /usr/lib/libxerces-c-3.0.so
  /usr/lib/libIL.so.1.0.0

Now if you ignore the references to the old compatibility libc.so.5 you can still find that there are quite a few programs that reinvent the wheel, reimplementing some functions that the C library already provides. Now, this would be fine and dandy if the implementation was subtly different, or was done with some particular purpose in mind, like glib’s functions, but I really can’t find the reason for the situation above to happen.

These are usually smaller functions, but still there is no reason for them to be present since anyway it’s more than likely that most of their use is replaced away by the compiler itself; more interesting is their presence in shared objects, since that would interpose around other calls, although this most likely won’t happen on glibc based systems since they provide versioned symbols which wouldn’t then interpose by default. That’s still a problem for *BSD systems though.

This has a much lower priority in my list than identifying all libraries bundled by various packages (even when they cannot be fixed, because they are proprietary or something else), because these are unlikely to turn out being security issues. On the other hand, the idea of doing such pointless duplication of common functions might as well caus security issues to be hidden, for instance if a package was to reimplement mktemp, then it would most likely be a problem.

Anyway for those interested to find out what’s duplicating code in their system, the new hit parade, derived directly from the Bug shows SQLite3 trying to climb up the ladder, together with boehm-gc. Why people can’t understand that there are libraries in the system already?

So after almost an year, I returned working on the library collision detection that comes with ruby-elf. I now have a much more powerful system to work on, but I also have a much bigger set of samples to scan, in my tinderbox.

Beside a few issues that I’ve had to taken care of, I’ve now got a database that contains a huge amount of data to process and useful information to derive. Just to give some statistics, the script harvested 24602 ELF files, counting 2713395 symbols, of which 326399 are duplicate between objects. These statistics have already counted out all the suppressed symbols known up to now, but obviously there are more yet unknown.

To stick with statistics for now, rather than actual results, it’s interesting to know that about 1365671 C++ ABI symbols, I sincerely wonder how many of these should be hidden instead.

On a more interesting note, Samba confirms itself sub-optimal by now having yet a convenience library for its shared functions, and copying over the symbols between the various pieces of the puzzle, included six different Python modules, whose total size would probably be cut in half I guess.

Sticking with the Python side, there is one damn issue that is really upsetting me: about thirty different packages link the Python interpreter statically rather than dynamically, resulting in around 30 different copies of Python itself in the full of Portage. Nasty. The problem is that the ebuild installs the shared and static libraries in two different paths, one of which being private “config” path for Python. The packages picking that up will explicitly request it at link time, causing Python to be linked in statically rather than dynamically:

Symbol PyBaseString_Type@@ (32-bit UNIX System V ABI Intel 80386) present 30 times
  /usr/games/bin/diameter
  /usr/bin/vim
  /usr/bin/gedit
  /usr/lib/root/libPyROOT.so.5.22
  /usr/lib/python2.5/site-packages/opencv/_highgui.so
  /usr/lib/libpython2.5.so.1.0
  /usr/bin/eog
  /usr/games/lib/mcl/plugins/python.so
  /usr/sbin/bextract
  /usr/bin/zapping
  /usr/lib/python2.5/site-packages/opencv/_cv.so
  /usr/bin/gvim
  /usr/bin/gwp
  /usr/bin/cooledit
  /usr/lib/planner/plugins/libpython-plugin.so
  /usr/sbin/bacula-fd
  /usr/sbin/bacula-sd
  /usr/lib/dia/libpython_plugin.so
  /usr/lib/xchat/plugins/python.so
  /usr/lib/xchat-gnome/plugins/python.so
  /usr/sbin/bacula-dir
  /usr/lib/gnumeric/1.8.3/plugins/python-loader/python_loader.so
  /usr/games/bin/adonthell
  /usr/games/bin/kiki
  /usr/lib/libgnt.so.0.0.0
  /usr/bin/epiphany
  /usr/lib/python2.5/site-packages/_lcms.so
  /usr/lib/perl5/vendor_perl/5.8.8/i686-linux/auto/Inline/Python/Python.so
  /usr/lib/apache2/modules/mod_wsgi.so
  /usr/lib/apache2/modules/mod_python.so

As you can see this includes two Apache modules, and quit a few pieces of GNOME. This is quite nasty. My suggestion until this is sorted out is not to enable python USE flag unless you really really really need it. The nastiest bit is that since there has been a Python vulnerability if you didn’t rebuild these packages after the bump, you’ll have them using a vulnerable interpreter, still. Do I really have to spell out how bad that is for stuff like Apache modules?

Short preamble: I’m in a very depressed mood, like I haven’t been in month; this is very bad for my health but usually means I can focus on things much better, so you might actually find out I’m doing more than usual. Of course there is also to count in that I’m working during holidays so it’s not going to be all nice at all, even counting my depression off.

As I’ve written, I don’t trust closed-source software even the slightest even though it does not really mean that free software is much better, process-wise, dealing with bundled libraries (like the bundled libs bug shows), with free software, or at least open-source software, there is the chance to check the sources out to fix the eventual issues.

This means that I won’t be using closed source software where security is a major concern, but since sometimes I have to use closed-source software, like Skype, or Sun’s compiler, it’s obvious that I have to find a compromise so I can still use them and yet feel reasonably safe. This is what is usually called having a mitigation strategy.

One of the most complex and well known mitigation strategies is of course SElinux, which makes a Linux system more like an APC than a computer. But such a system is probably safe to consider overkill for most systems, especially power user desktop systems.

Since this is, as I said, overkill, I’m more prone to look at smaller strategies, one of which I already discussed about: pam_mktemp . This module allows to create per-user private directories that make it much harder to exploit insecure temporary files vulnerabilities. Which is very nice since this seems to be a very common class of vulnerabilities, and my data shows that there is way too much software that still uses insecure functions to create temporary files, closed and open source alike.

Unfortunately, as you can read in my earlier blog post, this is not automatically a way out of the problem. The start-stop-daemon command from OpenRC plays nice with this just in the last release, and even with that, there are problems. The first problem is that the way pam_mktemp works, there is a need for the software calling PAM to open the session to properly set up the environment with its changes (which is what s-s-d lacked in previous versions). This causes for instance the gnome-keyring daemon to start with the wrong temporary directory when started by the PAM session chain. Even though pam_mktemp is invoked before the daemon, by the time it’s started the TMPDIR variable is not set in the environment. The reason for this is that the variable should not be changed if the session chain aborts the login.

The second problem is that not all software supports TMPDIR properly; Emacs has been fixed recently and now the emacs daemon starts up properly, but other software ignores TMPDIR altogether. VirtualBox (of which I still have things to say beside this) does not respect it for instance, which means that the module wouldn’t have spared you from the recent vulnerability that involved the software.

The third problem is that sometimes software expects TMPDIR to be world-readable, which is a bad assumption; Samba does this, and since s-s-d is now fixed, it now fails to work on my system. I still haven’t found out whether the PAM session chain was called at that point, and it’s just duplicating the problem with s-s-d with a different symptom, or if it fails to call it entirely. In either case, it’s a thing that has to be fixed to make sure that mitigation strategies like this one get in the default spirit of users.

But again this is just one part of the problem, and one part of mitigation. Other problems relate to the way we run some of the services, a lot of which still run as root rather than under a unprivileged user; while the git-daemon issue is now solved and the default install does not run as root any longer, there are more daemons that have the same problems.

Just as an example, I noticed that the iSCSI daemon ietd still runs under root, and I’ve added that to the list of software I have to check to see if I can improve it. Similarly, the init script for mpd does not use s-s-d to switch user but leaves it to mpd itself, spawning it by default with unneeded root privileges, and additionally not allowing pam_mktemp to create a new temporary directory for the mpd user (I have to spend some time on that since I’d also like to provide an alternative init script with multiplexing, which would then allow to run multiple mpds for different users, and in my case to just have the single mpd running as my own user rather than a different user entirely).

At any rate, I’m going to continue my best to make sure that secure defaults are in place in Gentoo, and that further mitigation strategies can be made available so that the users forced to use proprietary closed-source software don’t need to just accept whatever comes their way. Please join my efforts, if you can, by checking which software ignores TMPDIR and asking nicely upstream to fix the issue.

In my post regarding remote debugging (which I promised to finish with a second part, I just didn’t have time to test a couple of things), I’ve suggested the idea I’d like to have some kind of package splitting in Portage, to create multiple binary packages out of a single source package and ebuild, similarly to what distributions based on RPM or deb do (let’s call them RedHat and Debian, for historical reasons).

Now, I want to make sure nobody misunderstand me: I don’t intend to propose this as a way of removing the fine-grained control USE flags give us; I sincerely love that; and I also love not having to worry about installing -dev and -devel packages on my machines to be able to build software, even outside of the package manager’s control. I really find these two are strengths of Gentoo, rather than weakness, so I have no intention to fiddle with them. On the other hand, I think there are enough uses that would allow for an even finer control on binpkg level.

I’ve already given a scenario in my post about remote server debugging, but let’s try to show something different, something I’ve actually been thinking about myself. Yes I know this is a very vested interest for me, but I also think this is what makes Free Software great most of the time; we’re not solutions looking for problem, but usually solutions to problem one had at least at one point in time. Just like my writing support for NFS export on the HFS+ filesystem in Linux.

So let me try to introduce the scenario I’ve been thinking about. As it happens, I tend to a series of boxes in many offices for friends and friends of friends in my spare time, on the side. It’s not too bad, it does not pay my bills, but it does pay for some side things, which is good. Now since these offices usually use Windows, even though I obviously install Firefox as the second step after doing the system updates, it’s not unlikely that every other time I go there I have to clean up the systems. I think there are computers I’ve wiped up and reinstalled a few times already. I’ve now been thinking about setting up some firewalls based on Snort or similar. Since I am who I am, these would end up being Gentoo-based (as a side note, I’m tempted to set it up here so I can finally stop having trouble with Vista-based laptops that mess up my network). Oh and please, I know it might sound very stupid considering there are solutions good for this already, but considering how much I’m paid and the amount of money they are ready to spend (read: near to none), I would find it nicer to be paid to work on some Gentoo-related stuff than be paid to just look up and learn how to use already made equipment. Of course if you have suggestion, they are welcome anyway.

So anyway, in this situation I’d have to set up boxes that would usually feel very embedded-like: a common basis, the minimum maintenance possible, upgrades when needed. Donnie’s idea of using remote package fetching and instant deletion is not that good for this because it still requires a huge pipe to shove the data around; not only I don’t have so much upload bandwidth to employ for binpkging a whole system with debug information, it would also be a hit that most of my users wouldn’t like to have, on their bandwidth (if they want to use BitTorrent or look up p0rn from the office is not my problem).

With this in mind, I’d sincerely find it much nicer to be able to split packages, Portage-side, into multiple binary packages that can be fetched, synced, or whatever else, independently, as needed. As I proposed, a binpkg for the debug information files, but also a binpkg for documentation (including man and info pages), one for development data (headers, pkg-config), and maybe one for the prepared sources, that I want to talk about in a moment. With an environment variable it shouldn’t be much of a problem to choose which ones of these split binary packages to install in the system without explicit request; with a default including all of them but the debug informations and the sources. This would also replace the INSTALL_MASK approach as well as noinfo, noman, nodoc FEATURES. It wouldn’t be like a logical split of a package in multiple entries in the system, but rather a way to choose which parts to install, complementary to USE flags.

As for packaging the sources as I said above, there are two interesting points to be made for that, or maybe three. The first problem is that when you have to distribute a system based on Gentoo, you cannot just provide the binaries; since many packages are released under the GNU GPL version 2, even if you didn’t change the sources at all you should be distributing them alongside the binaries; and we modify a lot of sources. For license compliance we should also provide the full set of sources from which the code is derived. This is especially tricky for embedded systems. By packaging up the sources used for the builds, embedded distributors would be able to just provide all the -src subpackages as the full sources for the system.

The second point is that you can use the source packages for debugging too. Since there is, as far as I know,no way to fully embed the source code of software in the debug section of the files generated from that, the only way for GDB to display the source code lines during debugging is having the source files used for build available during the debugging session. This can easily be done by packaging up the sources and installing them in, say, /usr/src/portage/ when they are needed, from a subpackage.

A final point would be that by packaging sources in sub-packages, and distributing them, we could be reducing the overhead for users to unpack (maybe with uncommon package formats) and prepare sources (maybe with lots of patches and autotools rebuilding). Let’s say that every 6 hours a server produces md5-based source subpackages for all the ebuilds of the tree, or a subset of them. Users would then use those sources primarily, but still having the ebuilds to provide all the data and workflow so that the original untouched source would be enough to compile the package. Of course this would then require us to express dependencies on a per-phase basis, since then autotools wouldn’t be required at buildtime at all.

Okay I guess I’m really dreaming lately, but I think that throwing around some ideas is still better than not doing so, they can always be picked up and worked on; sometimes it worked.