Ned Ludd

Using mike's SuperH lanktak I've setup a new binrepo for those of you crazy enough to own one of these things or still have a dreamcast in the closet and a little time to spare.

It's painfully slow to compile on, but oh well..

machine : LANDISK
processor : 0
cpu family : sh4
cpu type : SH7751R
cpu flags : fpu ptea
cache type : split (harvard)
icache size : 16KiB (2-way)
dcache size : 32KiB (2-way)
bogomips : 266.24
master_clk : 266.66MHz
module_clk : 33.33MHz
bus_clk : 133.33MHz
cpu_clk : 266.66MHz
tmu0_clk : 8.33MHz


PROFILE: default-linux/sh/2006.1
PACKAGES: 371
ACCEPT_KEYWORDS: sh
CBUILD: sh4-unknown-linux-gnu
CHOST: sh4-unknown-linux-gnu
CFLAGS: -O2 -m4 -pipe
FEATURES: autoconfig buildpkg ccache distlocks metadata-transfer noauto sfperms splitdebug strict

Weeve @ gentoo hooked me up with an account on a Sun T2K and I've started a tinderbox run over there. The repo is pretty small in size right now (343 pkgs) but will be growing over time. I can see right now I've got lots of bugs to file and poking to do as this arch is lacking behind in stable keyword markings for a lot of standard packages in addition to lots of packages failing that are marked stable. But it will be a while before I can really dive head on into this as I'm moving next week.

Corey Shields over at Open Source Labs wrote an interesting peice on remote sysloging and hardened in his Life according to Corey blog.
It's a well written practical use of hardened in the real world. I'm quite pleased it's working out so well for OSL. I hope other major universities and the private sector can learn from their experiences.

Yesterday I got fwded along a forum posting over at penguinppc.org from our good friend over at the OSL [cshields@osuosl] powerpc-cpu optimizations

Gentoo PPC64 has glibc-2.3.4.20041102-r2 marked stable so first I started patching it up. While talking to another developer [vapier@gentoo] I found out that the powerpc-cpu optimizations had already been integrated in our glibc-2.4-r3 by him, so I stopped with the patching up of 2.3.x
Time for a few benchmarks. First nbench was not keyworded for PPC64 so I passed that info along to our PPC64 team and [ranger@gentoo] keyworded it for future use.

Here are the results.

Base PPC64 stable.
gcc-3.4.4 glibc-2.3.4.20041102-r2

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          581.52  :      14.91  :       4.90
STRING SORT         :            96.4  :      43.07  :       6.67
BITFIELD            :      1.2933e+08  :      22.19  :       4.63
FP EMULATION        :          36.512  :      17.52  :       4.04
FOURIER             :          8689.8  :       9.88  :       5.55
ASSIGNMENT          :          7.3297  :      27.89  :       7.23
IDEA                :          1503.4  :      22.99  :       6.83
HUFFMAN             :          589.62  :      16.35  :       5.22
NEURAL NET          :          14.411  :      23.15  :       9.74
LU DECOMPOSITION    :           556.8  :      28.85  :      20.83
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 22.153
FLOATING-POINT INDEX: 18.757
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 8 CPU
L2 Cache            : 
OS                  : Linux 2.6.5-7.97-pseries64
C compiler          : 3.4.4
libc                : 
MEMORY INDEX        : 6.069
INTEGER INDEX       : 5.154
FLOATING-POINT INDEX: 10.403
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

glibc-2.4.x requires gcc-4 so I compiled that, then recompiled nbench so we could establish any differences it makes alone.
gcc-4.1.1 with 2.3.4.20041102-r2

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          586.56  :      15.04  :       4.94
STRING SORT         :          102.48  :      45.79  :       7.09
BITFIELD            :      1.3137e+08  :      22.54  :       4.71
FP EMULATION        :          39.625  :      19.01  :       4.39
FOURIER             :            8742  :       9.94  :       5.58
ASSIGNMENT          :          10.188  :      38.77  :      10.06
IDEA                :          1750.9  :      26.78  :       7.95
HUFFMAN             :          775.66  :      21.51  :       6.87
NEURAL NET          :          16.845  :      27.06  :      11.38
LU DECOMPOSITION    :          605.04  :      31.34  :      22.63
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 25.276
FLOATING-POINT INDEX: 20.354
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 8 CPU
L2 Cache            : 
OS                  : Linux 2.6.5-7.97-pseries64
C compiler          : 4.1.1
libc                : 
MEMORY INDEX        : 6.948
INTEGER INDEX       : 5.866
FLOATING-POINT INDEX: 11.289
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Sadly the kernel version that's running on the OSL box is a SuSe one without the support needed. I kept getting FATAL: kernel too old while building glibc. Guess I'll have to save the powerpc-cpu optimizations testing for another day..

Recently thanks to the Oregon State University Open Source Labs I've been given access to an IBM OpenPower 720. This thing is a beast like no other box which I have access to. The specs are simply amazing. Anyway I noticed that Alec Warner/antarus@gentoo was having problems with running a cvs2git conversion of the gentoo-x86 tree, every box which he attempted it on ran out of memory. I figured ok well I've got access to the mothership and should not have any problems doing a run for him. We talked for a little while and he provided me with a quick little script to fire off the conversion process. Well it took 21 hrs consumed 100% of the CPU the entire time and then it failed, towards the end right before it died with an Out of Memory: Killed process 14671 (parsecvs) error. It had consumed 70.1G of virtual memory and 30G RSS as well as all the swap. The gentoo-x86 tree is about 1.4G worth cvs data, the parsecvs util had managed to convert that into 4.1G of git data before it got killed. Gotta say from an admin/infra point of view going from a 1.4G to +4.1G backend repo leaves little room to be desired.

None the less I don't see us switching to git any time soon unless the backend tools for conversion get a rewrite/update so they can process the full repo as incremental parts or learn how to use the existing memory more efficiently.

In this graph you can see where I started about at 21:00 and ran till about 18:00 the following day, at about 14:00 the basic conversion process was done and parsecvs started allocating memory here pretty quickly for another 4 hours. The final spike is when it started swapping to disk before it got killed.

Unfortunately the snmpd version running on the box does not appear to support 64bit counters so all the memory graphs are/were nil.

In the end I had fun helping him with this, and it really gave the power5 a workout
Sometime later this week I'll start setting up ppc64 binrepos, cross compilers etc..

Last weekend I got a list of packages from GMsoft(hppa lead dev) and started a big fat run on hake. Well it's pretty much finishing up and the only packages which I did not merge from his list of 1083 packages were those ones which were either ~arch or required a configured kernel (silly kde for having a harddep on media-sound/cdparanoia). Currently there are 758 pkgs in this repo and like the others, if/when something breaks I'll know about it and report it.

CFLAGS="-O2 -pipe -march=2.0"
CHOST="hppa2.0-unknown-linux-gnu"
CXXFLAGS="-O2 -pipe"
USE_ORDER=env:pkg:conf:defaults
PORTAGE_TMPFS="/dev/shm/portage"
PKGDIR=/packages/
MAKEOPTS="-j3"
CLEAN_DELAY=0
FEATURES="sandbox distclean buildpkg genpkgindex test collision-protect"
USE='dlloader test bindist'

Well I'm still busy doing whatever it is I do. Tim Yamin [plasmaro@gentoo] hooked me up with an account on the ia64 at OSL to get a repo going for it. So been that's pretty much what I've been doing all day and filing bugs for other packages and arches as I come across them on all the other hosts.

Current package count is 7011 across all repos and by the time I wake up hopefully it will be up it will be up by another 268. Then I got whatever is in this list unstable.ia64 to get keyworded and marked stable.

I do hope to get my hands on a fast mips host here in the near future so I can get that going as well. Hoping to break the 10k mark!!

jforman hooked me up with a new vhost for the tinderbox as the .x86. part is a bit deceptive. It's now just tinderbox.dev.gentoo.org for the master repo.

r2d2 is attempting to work with the x86 hardened binrepo directly to see if he can speed up the process of livecd creation. Added a few hundred more packages to the hardened/x86 repo for that making it the largest repo of them all.

Some point over the weekend Lance and me did a lot of work on the box that is going to be used to host most of the Google Summer of Code projects. Still got a few final touchup things todo on it, but thats mostly all just a bit of configuring services. It's still a bit to soon to tell if they are going to need more than 1 server for everything. genone's project might take a wee bit more space than the box we had allocated initially for this. One of the things I'd really like to see some of the money Gentoo will earn from doing this years SoC is that we pick up a nice server which can be used for future SoC events and or to host the fruitful project that come out of this years (like stats/anonsvn/anon-other or so). I do have high hopes and I've already picked out a pretty decent dell 2850 for about ~$4500 that I'd like to see us get that includes 12G Ram/dual dualcore xeons at 3GHZ each and about 100GB of space or so. (having a good server benefits everybody)

Cool new bug.. New QA warning - detect already stripped binaries in prepstrip (I like ELF)

The tinderbox current pkg count is at 6517 packages across 66 repos now. If anybody has a package or profile request to be added to the tinderbox please feel free to let me know.

default-linux/arm

I added another binary repo to the tinderbox. This time it's for the default-linux/arm/ profile. The build host is an arm netwinder and is so incredibly slow to build on it's not funny, (a PDA would be faster) but I gotta take one for the team if I want good coverage of all the profiles. This repo for sure wont be pure arch or ~arch either.

=================================================================
Processor       : StrongARM-110 rev 4 (v4l)
BogoMIPS        : 185.54
Features        : swp half 26bit fastmult 
CPU implementer : 0x44
CPU architecture: 4
CPU variant     : 0x0
CPU part        : 0xa10
CPU revision    : 4

Hardware        : Rebel-NetWinder
Revision        : 59ff
Serial          : 00000000000020f9
=================================================================
             total       used       free     shared    buffers     cached
Mem:           123         93         30          0          4         53
-/+ buffers/cache:         35         88
Swap:         1043         15       1028
=================================================================
Portage 2.1_rc1-r3 (default-linux/arm, gcc-3.4.6, glibc-2.3.6-r3, 2.6.14.2-grsec armv4l)
=================================================================

default-linux/mips/cobalt

I was starting the initial mipsel rsync repo transfer when the raq2.mips box locked up. I don't think anybody from the OSL will be able to reboot it anytime today so I'll probably delete whatever I have and wait for another day and another kernel. Boxes freezing up while simply transferring files is never good.

Icons
I noticed the icons on dev.gentoo.org were not loading after the migration to the new box so I updated those with the same eye candy ones I used on the tinderbox. Looks pretty sweet imo. Hopefully nobody will bitch about it being nice.

cross compiling kernels
Still on my todo list. I really don't have a clean way yet to map blindly which kernels I want to build using what options. Maybe next weekend.

So I pushed that pax-utils.

• Improved Makefile handling. (vapier)
• QA_TEXTREL/QA_EXECSTACK/QA_WX_LOAD exempt filtering flags. (kevquinn)
• Handle versioned symbols correctly on unstripped ELF files when using the -s flag.
• Do not assume it is ok to read from stdin if the -l -p flags were given. (solar)

Now it's time to get back to some cross compiling goodness. Well one thing I know about myself is anything I do I tend todo in excess. So I figure why build 1 cross compiler when you can build them all. Why merge into 1 place when you can reasonable test cross compiling for all arches.. Well that's just what I did and am doing.

Little script I'm using for just this task..

#!/bin/sh
# Copyright 2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: <solar@gentoo.org>

CFLAGS="-Os -pipe"
CBUILD=$(portageq envvar CHOST)
CXXFLAGS="${CFLAGS}"

export PYTHON_DONTCOMPILE=1

USE_SAVE="${USE}"
if [ ! -x /usr/bin/qlist ]; then
        echo "please emerge portage-utils"
        exit 1
fi

gcc_executables=$(qlist -oeCI gcc| sort -u | grep ^cross- | sed s/cross-//g | tr / -)

opts="$@"
for x in "$@"; do
        if [[ $x == "--libc" ]]; then
                opts="${opts/${x}/}"
                DO_LIBC=1
                do_libc=1
        fi
        if [[ "${lastopt}" == "--target" ]]; then
                ctarget_only="${x}"
                opts="${opts/${x}/}"
                opts="${opts/${lastopt}/}"
        fi
        lastopt=${x}
done

function cross_env_update() {
        mkdir -p ${ROOT}/etc
        for lpath in /usr/local/lib /usr/${CTARGET}/lib /usr/lib/gcc/${CTARGET}/$(${CTARGET}-gcc -dumpversion) /usr/lib/libstdc++-v3/; do
                echo "${lpath}" >> ${ROOT}/etc/ld.so.conf
        done
}

function cmerge() {
        CHOST=${1}
        shift

        export CTARGET=${CHOST}
        ARCH=$(echo ${CTARGET} | cut -d '-' -f 1)
        CC=${CTARGET}-gcc
        CXX=${CTARGET}-g++
        PKGDIR=${HOME}/packages/${CHOST}
        ROOT="${HOME}/ROOT/${CHOST}"
        LDFLAGS="-L${ROOT}/lib -L${ROOT}/usr/lib"

        do_libc=${DO_LIBC}
        unset ELIBC
        case ${CTARGET} in
                *-linux-gnu)    ELIBC=glibc;;
                *-linux-uclibc) ELIBC=uclibc;;
                avr) ELIBC=avr-libc;;
        esac
        export ELIBC
        [[ $ELIBC == "" ]] && do_libc=0

        USE="${ARCH} ${USE_SAVE}"
        case ${ARCH} in
                armeb) ARCH=arm;;
                s390x) ARCH=s390;;
                mips*) ARCH=mips;;
                powerpc64) ARCH=ppc64;;
                sparc*) ARCH=sparc;;
                powerpc) ARCH=ppc;;
                hppa*) ARCH=hppa;;
                sh4*|sheb) ARCH=sh;;
                x86_64) ARCH=amd64;;
                avr|ee|iop|cris) ARCH="x86";;
                i?86) ARCH=x86;;
        esac
        ACCEPT_KEYWORDS="${ARCH}"
        [[ "${unstable}" != "" ]] && ACCEPT_KEYWORDS="${ACCEPT_KEYWORDS}  ~${ARCH}"

        USE="-* ${ARCH} ${USE} elibc_${ELIBC} multicall make-symlinks"

        export ARCH ACCEPT_KEYWORDS CFLAGS CXXFLAGS LDFLAGS ROOT CBUILD CHOST CTARGET CC CXX PKGDIR USE

        mkdir -p ${ROOT}/etc
        if [[ $ELIBC != "" ]]; then
                if [ -e ${ROOT}/etc/ld.so.conf ]; then
                        if [ "$(md5sum ${ROOT}/etc/ld.so.conf|awk '{print $1}')" == "b18efa8c9f95b6aecc0974c0f54d8bb9" ]; then
                                touch ${ROOT}/etc/ld.so.conf
                                cross_env_update
                        fi
                else
                        cross_env_update
                fi
        fi

        if [[ "$do_libc" == 1 ]]; then
                emerge -b cross-${CHOST}/${ELIBC} ${opts}
        else
                emerge -b ${opts}
        fi

        unset ARCH ACCEPT_KEYWORDS LDFLAGS ROOT CHOST CTARGET CC CXX PKGDIR USE ELIBC
}

if [[ "$1" == "" ]]; then
        echo "$0: <opts> <emerge opts>"
        echo "   --libc"
        echo "   --target <chost>"
        echo "   --libc  --target i386-gentoo-linux-uclibc busybox -pv"
        exit 1
fi

for gnugcc in ${gcc_executables}; do
        [ -e /usr/bin/${gnugcc} ] || { echo bummer $gnugcc ; continue; }
        # include=$($gnugcc -print-file-name=)
        [[ $ctarget_only != "" ]] && [[ ${gnugcc/-gcc/} != $ctarget_only ]] && continue
        cmerge "${gnugcc/-gcc/}" "$@"
done

A blind run at building busybox and libc for just about every chost combo yields the following results. Anything marked N is something that failed. R means it installed of course.

tinderbox ~ # ./cmerge -qpv busybox
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/alpha-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/alpha-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/arm-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/arm-softfloat-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/arm-softfloat-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/arm-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/armeb-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/armeb-softfloat-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/armeb-softfloat-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/armeb-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/avr/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/cris-gentoo-linux-uclibc/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/ee/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/hppa-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/hppa1.1-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/hppa2.0-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/hppa64-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/i386-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/i386-pc-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/i686-gentoo-linux-uclibc/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/ia64-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/iop/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/m68k-gentoo-linux-uclibc/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/m68k-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/mips-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/mips-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/mips64-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/mips64el-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/mipsel-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/mipsel-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/powerpc-gentoo-linux-uclibc/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/powerpc-softfloat-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/powerpc-softfloat-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/powerpc-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/powerpc64-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/s390-ibm-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/s390x-ibm-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/sh-gentoo-linux-uclibc/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/sh-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/sh4-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/sh4-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/sh4eb-gentoo-linux-uclibc/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/sh4eb-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/sheb-gentoo-linux-uclibc/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/sheb-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/sparc-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/sparc-unknown-linux-gnu/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/sparc64-unknown-linux-gnu/ 
[ebuild  N    ] sys-apps/busybox-1.1.0 to /root/ROOT/x86_64-gentoo-linux-uclibc/ 
[ebuild   R   ] sys-apps/busybox-1.1.0 to /root/ROOT/x86_64-pc-linux-gnu/ 

Well that's cool and all. But how to test stuff? Thats where qemu comes to the rescue.. Few mins of hacking and poof we come up with a little something something like so.

#!/bin/sh

pkg_genext2fs() {

        if ! type -p genext2fs > /dev/null; then
                echo "no genext2fs"
                return 1
        fi

        cd $1 || return 1

rom=/rom

        mkdir -p {usr/,}{s,}bin etc/init.d proc/self dev/shm root tmp ./${rom}

        chmod 700 root
        chmod 1777 tmp

################
cat <<EOF > etc/init.d/rcS 
#!/bin/sh
rom=$rom
mount -a
mount tmpfs \$rom -t tmpfs
mkdir -p \$rom/proc \$rom/dev \$rom/ro \$rom/tmp
mount -o bind /dev \$rom/dev
mount -o bind /proc \$rom/proc
busybox cp -a bin etc lib root sbin usr \$rom/
chmod 1777 \$rom/tmp
pivot_root \$rom/ \$rom/ro
EOF
################

        chmod +x etc/init.d/rcS

        if [ ! -L proc/self/exe ]; then
                cd proc/self
                ln -s /bin/busybox exe

                if [ -e /proc/self/fd ] ; then
                        ln -fs /proc/self/fd fd 2> /dev/null
                        ln -fs fd/0 stdin
                        ln -fs fd/1 stdout
                        ln -fs fd/2 stderr
                fi
                cd ../../
        fi
        cd etc
        ln -sf ../proc/mounts mtab
        grep -v ^# /etc/protocols > protocols

        cd ..

        cd dev
        ln -sf /proc/kcore core

        # ttys
        for i in `seq 0 9`; do
                [ ! -e tty$i ] && mknod tty$i c 4 $i
        done
        [ ! -e tty ] && (mknod -m 666 tty c 5 0 2> /dev/null || echo cant mknod for tty)

        [ ! -e null ] &&  (mknod -m 666 null c 1 3 2> /dev/null || echo cant mknod for null)
        [ ! -e zero ] && (mknod -m 666 zero c 1 5 2> /dev/null || echo cant mknod for zero)
        [ ! -e urandom ] && (mknod -m 644 urandom c 1 9 2> /dev/null || echo cant mknod for urandom)
        [ ! -e random ] && mknod random c 1 8
        [ ! -e console ] && mknod console c 5 1
        [ ! -e full ] && mknod full c 1 7
        [ ! -e kmem ] && mknod kmem c 1 2
        [ ! -e mem ] && mknod mem c 1 1
        [ ! -e port ] && mknod port c 1 4

        # IDE devs
                [ ! -e hda ] && mknod hda b 3 0
        [ ! -e hdb ] && mknod hdb b 3 64
        [ ! -e hdc ] && mknod hdc b 22 0
        [ ! -e hdd ] && mknod hdd b 22 64

        # loop devs
        for i in `seq 0 7`; do [ ! -e loop$i ] && mknod loop$i b 7 $i; done

        # ram devs
        for i in `seq 0 9`; do [ ! -e  ram$i ] && mknod ram$i b 1 $i ; done
        [ ! -L ram ] && ln -s ram1 ram

        # virtual console screen devs
        for i in `seq 0 9`; do [ ! -e vcs$i ] && mknod vcs$i b 7 $i; done

        chown root:tty tty 2> /dev/null
        chown root:sys null zero 2> /dev/null
        cd -

################
## /etc/fstab ##
################
cat <<EOF > etc/fstab
#/dev/root      /               ro,defaults  defaults 0 0
proc            /proc           proc    defaults 0 0
#none           /dev            devfs   defaults 0 0
devpts          /dev/pts        devpts  defaults,gid=5,mode=620  0 0
tmpfs           /tmp            tmpfs   defaults 0 0
tmpfs           $rom            tmpfs   defaults 0 0
tmpfs           /dev/shm        tmpfs   defaults 0 0
EOF
################

        cd ${TOPDIR} || return 1
        cd $1 || return 1

        scanelf -qnBR . | grep libgcc
        REALSIZE=$(LANG=C du ./ -s -c -k | grep total | awk '{print $1}')
        ADDTOROOTSIZE=$([ $REALSIZE -ge 20000 ] && echo 16384 || echo 16)
        SIZE=$(( $REALSIZE + $ADDTOROOTSIZE ))
        INODES=$(($(find ${DESTDIR} | wc -l) + 400))
        genext2fs -i ${INODES} -b ${SIZE} -q -d . $2
        cd $TOPDIR
}

TOPDIR=$PWD

tarballs="$@"
[[ $1 == "" ]] && tarballs="$(ls *.tar.bz2)"

for x in ${tarballs} ; do
        d=$(basename $x .tar.bz2)
        cd $TOPDIR
        pkg_genext2fs ${d} $TOPDIR/${d}.ext2
        [ $? != 0 ] && echo WTF $?
done

End result..
arm-gentoo-linux-uclibc.ext2 armeb-softfloat-linux-uclibc.ext2 mips-gentoo-linux-uclibc.ext2 powerpc-softfloat-linux-uclibc.ext2
arm-softfloat-linux-uclibc.ext2 i386-gentoo-linux-uclibc.ext2 mipsel-gentoo-linux-uclibc.ext2 sh4-gentoo-linux-uclibc.ext2
armeb-gentoo-linux-uclibc.ext2 i686-gentoo-linux-uclibc.ext2 powerpc-gentoo-linux-uclibc.ext2

As we can see uClibc is clearly better at everything. In this case being cross compiled and producing all the parts needed in order to be booted..

wget http://tinderbox.x86.dev.gentoo.org/portage/local/misc/i386-gentoo-linux-uclibc.ext2

qemu -kernel /boot/bzImage -append "root=/dev/hda init=/sbin/init rootfstype=ext2 ro" i386-gentoo-linux-uclibc.ext2

Sweet worked like a charm..
Next task is to mass x-compile some kernels for other arches and see what qemu can do for us..

So it's probably going to be a while before we can do drive swapping on for pitr/dustpuppy and dustpuppy is offline for till that happens so I've started a stable multilib hardened repo for amd64 using an chroot on pitr. hardened/amd64/multilib/
The repo is just shy of about 500 of the most common packages now.

It's make.conf

CFLAGS="-O2 -pipe -fforce-addr"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
USE="-nls -tcpd userlocales dlloader snmp bzip2 cli cgi session tiff jpeg png sysfs bindist boundschecking mp3 ogg vorbis png vcd mpeg xml xml2"
MAKEOPTS="-j4 --quiet"
FEATURES="buildpkg distclean nodoc noinfo"
PKGDIR=/usr/portage/local/hardened-multilib
CLEAN_DELAY=0
USE_ORDER=env:pkg:conf:defaults

Added quite a few more packages to the hardened/ppc repo. Current count there is 468

Per request I'm starting a ppc-uclibc repo. I started from a stage1-ppc-uclibc-2005.0 but am running into a few problems. First every single package wanted to install itself into a new SLOT. This was caused because no SLOT file existed in that stageball. Then while running $PORTDIR/scripts/bootstrap.sh uclibc itself would puke saying that the old version was built with +nls and I needed to keep it enabled. Well that's bogus because gentoo nls support is/was non exisxtant back then for uclibc. The problem seems to be that the built_with_use function when no USE= file exists returns an incorrect value.

While updating stuff I noticed that the default icon sets for apache are really ugly. I asked around and sure enough before long somebody pointed me in the right direction for some sweet eye candy.

Pushed a new portage-utils (0.1.17) the other day.
ChangeLog

• q: Updated stderr/stdout handling for BSD again.
• qfile: do not abort when user passes qfile "..."
• qimlate: New applet (Thomas Cort tcort@gentoo.org) uses portage metadata/cache directly.
• qmerge: bug fix. dont remove vdb entries in pretend mode.
• qpkg: new switch -P/--pkgdir to allow user defined pkgdirs.
• quse: new switch. -N/--name-only used to only display matching entries and not the values.

If vapier has not pushed a new pax-utils already then I'll try to get to that in the next few days. It's got a few updates to make life suck less.

Well thanks to Corey Shields of OSU OSL our tinderbox now has 167 GB more space to play with. I've already moved all the web/ftp content over to that drive.
We also enabled rsync as a service on the box so that anybody using the binrepo as the PORTAGE_BINHOST= for new installs can have the same tree used to build stuff. Any customizations (rare but needed here and there) are in $PORTDIR/local dir but that of course excluded by default when doing an emerge --sync.

Kevin Quinn hacked on pax-utils the other day and managed to come up with a patch that shifts all the QA_TEXTREL/QA_EXECSTACK filtering logic into the scanelf utility directly. This is good news to us because it keeps ebuild.sh clean and all the logic in a single place between the stable and unstable portage trees.

portage-2.0.54-r2 is starting to be marked stable by the arches. This is also good news as it fixes a silly long standing emerge -e portage looping bug with minor revisions.

I did a cvs -Q up yesterday and noticed I had forgotten to commit a netkit-telnetd update from a while ago. Our netkit-telnetd package inherits patches from debian. While I was checking to see what the latest debian patch level was I became inspired to do something similar to their pkg listing page. debian netkit-telnet

So I started hacking on a little bash script last night before I went to sleep,when I woke up today I started hacking on it a bit more and it's starting to look ok now. I still want to gather and display a bit more information than I am now, but I'm starting to become inspired to eat lunch and do nothing for a few hours. So here's what i've got now.. Gentoo netkit-telnetd

The index pages

• hardened/ppc
• hardened/x86
• default-linux/x86
• default-linux/m68k
• default-linux/armel
• uclibc/i386
• uclibc/armeb-softfloat
• uclibc/armeb-hardfloat
• uclibc/sh4

Hopefully somebody else will come along and do it the right way. IE full .xml/.rss feeds and linking it to http://packages.gentoo.org etc..

Our tinderbox now has 5259 binpkgs between all of it's $CHOST repos. (not bad.. glep19 stuff almost)

Managed to get portage-2.0.54-r2 out the door yesterday. Which is cool because it fixes an annoying looping bug with -e system that's been present for a long time (but overlooked in 2.0.x). SpanKY and Kevin Quinn are trying to perfect a small snippet of code for handling text relocations and executable stacks. I expect that we will see a portage-2.0.54-r3 after those two guys work it out, which hopefully will make it a candidate for stable marking.

TclPython is neat but giving me a few problems with stdout handling that it really should not be giving me. Example: if {[file exists /usr/lib/tclpython/tclpython.so ]} { catch {load /usr/lib/tclpython/tclpython.so} } package require tclpython 4 set interp [python::interp new] $interp exec {import site, sys} $interp exec {sys.path.append("/var/irc/jeeves")} $interp exec {import metadata} $interp exec {sys.argv=['e','mtd']} catch { [$interp exec {metadata.main()}] } result python::interp delete $interp The metadata.py can be found at http://people.gentoo.org/solar/irc/metadata.py Anybody that's worked with Tcl knows the catch {} should grab the stdout/stderr and shove the output in $result. However it's printing directly to the console and never getting saved in $result. major bummer..

I just discovered tclpython yesterday. I really did not have time to play with it then, but I'm trying to squeeze in playing with it while sitting here at work and updating/renewing a few hundred domains.
If all goes well I hope to speed up the bot known as jeeves when doing !herd and !metadata lookups on irc by keeping the python interpreter loaded in memory. Oh and it's an unmaintained package in portage in need of a bump to 4.1.
Guess I'll do that.

For a few months now I've profile testing on pretty decent x86 host at OSU. As a side result of building and leaving FEATURES=buildpkg enabled I started ending up with pretty decent sized binary repositories which was pretty cool. I moved on to expanding it to make other arch test/tinderboxes start pushing the $PKGDIR from their local repos as well. Also as a result of testing I crontab a crossdev job to build nearly every known cross-compiler on the face of the planet that we could possibly support. As soon as KingTaco and I can do some disk shuffling at OSU I'll have an amd64 to beat on in the same way as the x86 tinderbox. Currently I'm limited to profile testing on mipsel/arm/ppc/x86/hppa. x86 has the best coverage of them.

Side note I wanted to get portage-2.0.54-r2 out the door yesterday but that got delayed. Today is a bad day for me and development so I don't think I'll be pushing one today either.