Running with compressed kernelmodules

[technosaurus]

yes you have come across another of the many programs that is riddled with "GNU=isms" It would probably need getline instead of gnu_getline (may need to change the constructs a bit) ... comment out getdelim in the find code so that it uses the diet version and define the types (pid_t and time_t) with their corresponding types such as:
#define pid_t int //maybe unsigned int?

As you can see, it takes some time and work to get things to work with dietlibc (including making use of libowfat sometimes)

see my efforts here:
http://code.google.com/p/diet-binaries/downloads/list

[amigo]

Karl, you need to get your CFLAGS all together like this:
CFLAGS="-ffunction-sections -fdata-sections -Os -D _BSD_SOURCE"
Otherwise only the last part is being set.

[Karl Godt]

Finally I got me a find-4.1.20 of 92kb (/1.024 /1.024 = 87kb) :

Code: Select all

./configure LDFLAGS="-static -Wl,--gc-sections,--as-needed" CFLAGS="-ffunction-sections -fdata-sections -Os -D _BSD_SOURCE" CC="diet gcc -nostdinc"

- which behaves like before -

l10nflist.c:(.text._nl_make_l10nflist+0x1b0): undefined reference to `stpcpy'
l10nflist.c:(.text._nl_make_l10nflist+0x1cc): undefined reference to `stpcpy'
../intl/libintl.a(l10nflist.o):l10nflist.c:(.text._nl_make_l10nflist+0x1f8): more undefined references to `stpcpy' follow
collect2: ld returned 1 exit status
make[3]: *** [xargs] Error 1
make[3]: Leaving directory `/COMPILE/FINDUTILS/findutils-4.1.20/xargs'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/COMPILE/FINDUTILS/findutils-4.1.20/xargs'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/COMPILE/FINDUTILS/findutils-4.1.20'
make: *** [all] Error 2
23:16 findutils-4.1.20 # strip /COMPILE/FINDUTILS/findutils-4.1.20/find/find
23:25 findutils-4.1.20 # ls -s /COMPILE/FINDUTILS/findutils-4.1.20/find/find
92 /COMPILE/FINDUTILS/findutils-4.1.20/find/find
23:26 findutils-4.1.20 # file /COMPILE/FINDUTILS/findutils-4.1.20/find/find
/COMPILE/FINDUTILS/findutils-4.1.20/find/find: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, stripped
23:31 findutils-4.1.20 #

with outcommented lines for "stpcpy" at getline.h+.c , l10nflist.c , dcigettext.c and for "mbrtowc" at quotearg.c
This find is working though, but locate and xargs did not compile (as quoted above).
And find is still larger than the original initrd.gz-find.

Further it turned out that busybox more and less do not work in the rdsh like they use to at the prompt or in X normally : they rush to the very end of the file. e3 is already a very sophisticated and good editor but e3vi ( trying to exit e3vi with Ctrl-z) gave me a hang and the kernel did not reboot though i passed a panic=40 param to the grub commandline.
The busybox modinfo also needs two tweaks in the init-script :

Code: Select all

[ "`modinfo aufs | grep vermagic 2>/dev/null`" = "" ]
[ "`modinfo ppa | grep vermagic 2>/dev/null`" != "" ]

because busybox modinfo does not produce an error exit code if the module is not found .
Also the commands reboot,poweroff and halt do not work simply typed at the commandline. I'tll try again with umount -ar , fusermount -z -u , umount-FULL -i -n -l , kill 0 -3 -9 as it is mentioned at rc.shutdown.
waitmax and fsck are the only binaries that are not used by the scripts (e2fsck bianry is used) and bb-fsck hasn't got ext4 support.

Btw : Which version control system and licence to choose at code.google ? I have a few on skydrive at live/hotmail but to provide source for eg. kernels 50mb are not enough ( splitted into drivers and else ) and the navigation there is quite uncomfortable.

[PANZERKOPF]

And find is still larger than the original initrd.gz-find.

Because initrd's find is compressed with UPX.
e2fsck, resize2fs,fsck also are compressed with UPX.
You can see an UPX signature at offset 078h-07Ah.

[technosaurus]

for stpcpy try adding -lcompat to the end of your LDFLAGS and or this in the includes of the referenced .c file
#include <string.h>

for mbrtowc try adding this to the list of includes
#include "dietlocale.h" //maybe
#include <wchar.h> //for sure
#include <errno.h> //maybe

maybe try this busybox patch pack that I cobbled together to use the old full e2fsprogs from various sources using chicken wire and duct tape.

Notes:
It has some compile warnings, but worked ok
Needs updated to new busybox constructs and ext4 support
I didn't add in the full usage text

[Karl Godt]

maybe try this busybox patch pack that I put together to use the old full e2fsprogs

Already wondered how to enable them at the menuconfig .
Think I've found it :

@@ -469,6 +469,10 @@
coreutils/ \
coreutils/libcoreutils/ \
debianutils/ \
+ e2fsprogs/e2p/ \
+ e2fsprogs/ext2fs/ \
+ e2fsprogs/blkid/ \
+ e2fsprogs/uuid/ \
e2fsprogs/ \
editors/ \
findutils/

will take me a while to test it .. there are already 17 _install inside the source dir ... I think I've to delete halv of them ... even some DOTconfigs .... I am getting confused

Because initrd's find is compressed with UPX.

Next interesting news :
http://upx.sourceforge.net/

upx EXECUTABLE seems to be working pretty good
87kb > 46kb

for stpcpy try adding -lcompat to the end of your LDFLAGS and or this in the includes of the referenced .c file

OK : I'll start that again in 10 hours ... need some sleep now .

[Karl Godt]

Code: Select all

./configure LDFLAGS="-static -Wl,--gc-sections,--as-needed -lcompat" CFLAGS="-ffunction-sections -fdata-sections -Os -D _BSD_SOURCE" CC="diet gcc -nostdinc"

and adding #include <string.h> did not help . <string.h> had been only [missing] at getline.c+.h . This time I commented

Code: Select all

#ifdef _GNU_SOURCE
int vasprintf(char **strp, const char *fmt, va_list ap);
// for FIDUTILS // ssize_t getline(char **lineptr, size_t *n, FILE *stream);
// for FIDUTILS // ssize_t getdelim(char **lineptr, size_t *n, int delim, FILE *stream);
#endif

at dietlibc/include/stdio.h and commented the static definitions for stpcpy : Find did compile same size 122/87/46kb (staticdietlibc/strip/upx -7) , xargs and locate not

l10nflist.c:(.text._nl_make_l10nflist+0x1cc): undefined reference to `stpcpy'
../intl/libintl.a(l10nflist.o):l10nflist.c:(.text._nl_make_l10nflist+0x1f8): more undefined references to `stpcpy' follow
collect2: ld returned 1 exit status
make[3]: *** [xargs] Error 1
make[3]: Leaving directory `/COMPILE/FINDUTILS/findutils-4.1.20/xargs'
make[2]: *** [all-recursive] Error 1

I made a "fancy-full-blown--long-options-busybox" of statically 844kb and with lzma compressed drivers the initrd.gz became 1363kb . But busybox less and more seem to need nvidiafb or intel-agp support to work like they normally do even at the prompt. ed is described at the menuconfig from year 1970 and "really evil" and somehow it's true : working, but uncomfortable.
bb modprobe seems to ignore module.alias even if support for modules.alias is enabled ( perhaps just for depmod )

Code: Select all

MODALIASES="`cat /sys/bus/pci/devices/*/modalias`" #important, save to variable before loop.
for ONEMODALIAS in $MODALIASES
do
  modprobe -v $ONEMODALIAS #-v means verbose.
  [ $? -eq 0 ] && echo -n "." > /dev/console
done

for ONEHID in `grep -o 'MODALIAS=hid:[^ ]*' /tmp/uevents.log | cut -f 2 -d '=' | tr '\n' ' '`
 do
  modprobe $ONEHID
 done

modprobe: module pci:v000010DEd00000182sv0000107Dsd00002922bc03sc00i00 not found in modules.dep
modprobe: module hid:b0003v0000413Cp00002010 not found in modules.dep

One good new : busybox eject is working with creating a symlink to /dev/sr0 from /dev/cdrom .
Booting time : 10 seconds for loading drivers.ko.lzma , +18 seconds for searching puppy files (500GB/12-83/4-82 partitions) , +45 seconds after [Enter] to load the pupsave-2fs and drivers.ko for reaching jwm .

And this is busybox-linker.sh :

Code: Select all

#!/bin/bash
cd `pwd`
./busybox > bbcommands
cat bbcommands | sed '/^$/d' > bbcommands2
BEGIN=`cat -n bbcommands2 | grep Currently | tr -s " " | tr "\t" " " | cut -f 2 -d " "`
LINES=`cat -n bbcommands2 | wc -l` 
COMMANDBLOCKLINES=`expr $LINES - $BEGIN`
COMMANDBLOCK=`cat bbcommands2 | tail -n $COMMANDBLOCKLINES | tr ',' ' ' | tr -s " " | tr " " "\n" | tr "\t" " " | tr -d " " | sed '/^$/d'`
for ONELINE in $COMMANDBLOCK ; do
ln -s busybox $ONELINE 
done
rm bbcommands
rm bbcommands2

and bb-unlinker.sh :

Code: Select all

#!/bin/bash
cd `pwd`
file * | grep symbolic > links 
cat links | grep busybox > busyboxlinks
cat busyboxlinks | cut -f 1 -d ':' >busyboxlinks2
BBLINKS=`cat busyboxlinks2`
for ONELINE in "$BBLINKS" ;do
rm $ONELINE
done
rm links
rm busyboxlinks
rm busyboxlinks2

[technosaurus]

I made a "fancy-full-blown--long-options-busybox" of statically 844kb and with lzma compressed drivers the initrd.gz became 1363kb . But busybox less and more seem to need nvidiafb or intel-agp support to work like they normally do even at the prompt. ed is described at the menuconfig from year 1970 and "really evil" and somehow it's true : working, but uncomfortable.
bb modprobe seems to ignore module.alias even if support for modules.alias is enabled ( perhaps just for depmod )

Even if you do allyesconfig, you still need to manually enable full modutils - otherwise it uses the small version which operates differently and thus uses bb.* prefix

Re: look of more/less - I thought Barry's kernels were compiled with vesafb only ... I need to check that, b/c it had prevented me from testing some stuff before.

by the way there is a new feature that wasn't documented very well:

Code: Select all

busybox --list

I use this to place all of the links in /usr/local/bin so that if the full version exists, it gets used instead(due to the $PATH precedence)

Code: Select all

for x in `busybox --list`; do ln -s busybox $x;done

[Karl Godt]

busybox --list

Really works at 1.17.1 compiled statically and dynamicallly , and it seems with and without --long-options enabled .
The puppy bb-1.15.0 doesn't have it .

bb `umount -l` works nicely at the rdsh to unmount everything one after another for doing the filesystemcheck .
bb `reboot -f` worked , too.
bb `lspci -k` could replace `elspci -l` .

[PANZERKOPF]

Re: look of more/less - I thought Barry's kernels were compiled with vesafb only

Barry's kernels were compiled with vesafb but without console framebuffer. When we boot using vga=??? we get a black screen ( we should blindly type "modprobe fbcon" to get a working console.)
Barry's config:
CONFIG_FRAMEBUFFER_CONSOLE=m
Should be:
CONFIG_FRAMEBUFFER_CONSOLE=y

[technosaurus]

Barry's kernels were compiled with vesafb but without console framebuffer. When we boot using vga=??? we get a black screen ( we should blindly type "modprobe fbcon" to get a working console.)
Barry's config:
CONFIG_FRAMEBUFFER_CONSOLE=m
Should be:
CONFIG_FRAMEBUFFER_CONSOLE=y

ahh - so that may be why I could never get SDL-fb, DirectFB, QT-embedded,gtk2-fb, fbui and others to work?

forgot to mention that waitmax can be replace with busybox timeout with a very minor rewrite of the scripts using waitmax

2 more busybox addons (work but still need some work - see notes)

[Karl Godt]

Kernel 2.6.30 is really tricky with framebuffer and video drivers . Until now I've compiled .6-.9 at least 11 times and only 2 are working ok with my hardware to show the linux-penguin . This has also additionally the older IDE drives support enabled.

[PANZERKOPF]

ahh - so that may be why I could never get SDL-fb, DirectFB, QT-embedded,gtk2-fb, fbui and others to work?

This may be a reason because each of them needs framebuffer.
I attached my configuration of Barry's kernel 2.6.33.2 used in Quirky and other Puppyes. Framebuffer console is enabled.

[Karl Godt]

I got `less` and `more` to work in the rdsh :

It needs bb-getty [ and login ] with more than /etc/shadow [ packed group,gshadow,passwd also into it ] .

getty 38400 tty[1-4] changes the output of `tty` from /dev/console to /dev/tty[1-4] and `less/more` are working correctly.
They do not work on '/dev/tty0' though I added /dev/tty and /dev/tty0 to the initrd-archive [ also /dev/video /dev/video0 /dev/fb1 ] .

[Karl Godt]

It seems that bb-lspci is not able to replace elspci because it shows [ like lspci-FULL ] only the 4 middle chars of /sys/bus/pci/devices/*/class , while elspci shows the last 6 chars .

#w001 NOTE: in 4.1.2 I use elspci to determine if yenta-socket needs to be loaded.
#if so, need a delay -- i used sleep 2 -- before elspci can be used to detect 0C0310
#pci interface -- in the case of my pccard-usb adaptor, which needs ohci-hcd.
#my new code below will need to be fixed for that situation...

#w003 maybe this will fix... look for 'bc0Csc03i10' in modalias...
MODALIASES="`cat /sys/bus/pci/devices/*/modalias`" #important, save to variable before loop.
if [ "`echo "$MODALIASES" | grep 'bc0Csc03i10'`" != "" ];then
modprobe yenta-socket
[ $? -eq 0 ] && sleep 2
fi

#v423 k2.6.29.6 dmesg has warning that ehci-hcd should load before uhci-hcd and ohci-hid
# this is contrary to my understanding all these years, but oh well, let's do it...
#ELSPCI="`elspci -l`" #jesses great little utility.
#[ "`echo "$ELSPCI" | grep '0C0320'`" != "" ] && modprobe ehci-hcd

Code: Select all

#[ "`echo "$ELSPCI" | grep '0C0310'`" != "" ] && modprobe ohci-hcd
#[ "`echo "$ELSPCI" | grep '0C0300'`" != "" ] && modprobe uhci-hcd

These two sniplets could replace elspci in the initrd.gz :

Code: Select all

PCICLASS="`cat /sys/bus/pci/devices/*/class`"
if [ "`echo "$PCICLASS" | grep '0x0c0320'`" != "" ];then
 modprobe ehci-hcd
 [ $? -eq 0 ] && sleep 2
fi

or

Code: Select all

#CURDIR=`pwd`
#cd /sys/bus/pci/devices
#find * -follow -maxdepth 5 -name class -exec realpath {} \; 2>/dev/null > /tmp/FIND-CLASS ## needs bb-realpath enabled
#F=`cat /tmp/FIND-CLASS | sort -u`
#if [ "`for OF in "$F"; do grep '0x0c0320' $OF;done`" != "" ]; then modprobe ehci-hcd; fi
## rm /tmp/FIND-CLASS
#cd "$CURDIR"

I had a look at the source of bb-lspci and elspci but have no idea how to enable 6/8 char output .

[Karl Godt]

bb modprobe seems to ignore module.alias even if support for modules.alias is enabled ( perhaps just for depmod )

I found out, that this is fortunately not true
Unfortunately it seems that every mainboard has got different additional modaliases which are not listed in modules.alias .
So it looks in the .alias file but only mentions that it cannot find it in the .dep file .
I tested it by restoring the dirs /lib/modules/`uname -r`/* and copying the uhci-hcd module back from /pup_new/lib/* into it , running depmod , adding a PS2-keyboard to the USB-keyboard and modprobe [ -r ] uhci-hcd / modprobe pci:v*d*sv*sd*bc0Csc03i00* : this both worked to switch the USB-keyboard on/off .

Another bb-feature to try is `bootchartd start` but it presents three large log files with somehow unuseable data :
[/var/log/]proc_diskstats.log : after several minute > 13,000 lines of

9174
2 0 fd0 0 0 0 0 0 0 0 0 0 0 0
2 1 fd1 0 0 0 0 0 0 0 0 0 0 0
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
1 2 ram2 0 0 0 0 0 0 0 0 0 0 0
1 3 ram3 0 0 0 0 0 0 0 0 0 0 0
1 4 ram4 0 0 0 0 0 0 0 0 0 0 0
1 5 ram5 0 0 0 0 0 0 0 0 0 0 0
1 6 ram6 0 0 0 0 0 0 0 0 0 0 0
1 7 ram7 0 0 0 0 0 0 0 0 0 0 0
1 8 ram8 0 0 0 0 0 0 0 0 0 0 0
1 9 ram9 0 0 0 0 0 0 0 0 0 0 0
1 10 ram10 0 0 0 0 0 0 0 0 0 0 0
1 11 ram11 0 0 0 0 0 0 0 0 0 0 0
1 12 ram12 0 0 0 0 0 0 0 0 0 0 0
1 13 ram13 0 0 0 0 0 0 0 0 0 0 0
1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
7 0 loop0 0 0 0 0 0 0 0 0 0 0 0
7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
7 2 loop2 0 0 0 0 0 0 0 0 0 0 0
7 3 loop3 0 0 0 0 0 0 0 0 0 0 0
7 4 loop4 0 0 0 0 0 0 0 0 0 0 0
7 5 loop5 0 0 0 0 0 0 0 0 0 0 0
7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
8 0 sda 2040 445 17254 7196 44 0 448 24 0 7192 7220
8 1 sda1 182 31 1474 536 2 0 16 0 0 536 536
8 2 sda2 3 0 18 0 0 0 0 0 0 0 0
8 3 sda3 232 10 1898 872 2 0 16 0 0 872 872
8 4 sda4 3 31 48 8 0 0 0 0 0 8 8
8 5 sda5 573 7 4972 2000 28 0 320 16 0 1992 2016
8 6 sda6 211 18 1730 576 2 0 16 0 0 576 576
8 7 sda7 3 15 48 16 0 0 0 0 0 16 16
8 8 sda8 306 7 2466 768 2 0 16 4 0 772 772
8 9 sda9 239 34 1954 1336 2 0 16 4 0 1340 1340
8 10 sda10 223 18 1826 628 2 0 16 0 0 628 628
8 11 sda11 12 31 114 36 2 0 16 0 0 36 36
8 12 sda12 17 18 178 112 2 0 16 0 0 112 112
8 13 sda13 5 99 104 36 0 0 0 0 0 32 36
8 14 sda14 7 99 120 32 0 0 0 0 0 32 32
8 15 sda15 7 21 120 28 0 0 0 0 0 28 28
259 0 sda16 0 0 0 0 0 0 0 0 0 0 0
11 0 sr0 5 0 20 88 0 0 0 0 0 88 88

proc_ps.log : > 10,000 lines
9174

1 (sh) S 0 0 0 0 -1 4202752 3870 270133 0 0 10 104 1476 379 20 0 1 0 2 1241088 256 4294967295 12587008 13072608 3216404416 3216398668 134773413 0 0 16388 134217730 0 0 0 0 0 0 0 0 0 0
2 (kthreadd) S 0 0 0 0 -1 2149621824 0 0 0 0 0 0 0 0 15 -5 1 0 2 0 0 4294967295 0 0 0 0 0 0 0 2147483647 0 0 0 0 0 0 0 0 0 0 0

2165 (loop4) S 2 0 0 0 -1 2149621824 0 0 0 0 0 0 0 0 0 -20 1 0 2879 0 0 4294967295 0 0 0 0 0 0 0 2147483647 0 0 0 0 17 0 0 0 0 0 0
2245 (syslogd) S 1 2245 2245 0 -1 4202560 8 0 0 0 0 2 0 0 20 0 1 0 3077 1236992 250 4294967295 12587008 13072608 3216062592 3216057728 134521406 0 0 5 16386 0 0 0 17 0 0 0 0 0 0
2247 (klogd) S 1 2247 2247 0 -1 4202816 11 0 0 0 1 2 0 0 20 0 1 0 3084 1236992 253 4294967295 12587008 13072608 3213436064 3213431248 134522916 0 0 1 58751526 0 0 0 17 0 0 0 0 0 0
2248 (bootchartd) R 1 0 0 0 -1 4202816 11 0 0 0 0 0 0 0 20 0 1 0 3091 1245184 255 4294967295 12587008 13072608 3213270112 3213260572 134521406 0 0 0 18951 0 0 0 17 0 0 0 0 0 0

proc_stat.log > 2,900 lines

9174
cpu 1488 0 504 6525 641 6 8 0 0
cpu0 1488 0 504 6525 641 6 8 0 0
intr 25680 71 8 0 0 0 0 3 0 0 0 0 525 0 0 2090 66 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 16612
btime 1296654903
processes 2251
procs_running 1
procs_blocked 0

and a header file of 4 lines

version = 0.8
title = Boot chart for (none) (Wed Feb 2 13:56:34 UTC 2011)
system.uname = Linux 2.6.30.5 #1 SMP Tue Sep 1 15:48:26 GMT-8 2009 i686
system.kernel.options = psubdir=JUMP-5 pdev=sdb5 vga=normal debug panic=40 pfix=rdsh

whose data is found also at /tmp/bootkernel.log &| /var/log/messages .
bootchartd also needs tar enabled ( produces a /var/log/bootchart.tgz ) .
So these both (bootchartd/tar) can be disabled with good awareness !

[Karl Godt]

About running `init` at the rdsh :

Code: Select all

#RDSH is a boot param. exit to initial ramdisk shell...
if [ "$RDSH" = "yes" ];then
 [ "` which syslogd`" != "" ] && syslogd &
 [ "`which klogd`" != "" ] && klogd &
 [ "` which bootchartd`" != "" ] && [ "`which tar `" != "" ] && bootchartd start &
 echo > /dev/console
 echo "Dropped to initramfs shell. Type 'exec switch' to continue booting Puppy." > /dev/console
 exec init
 echo `ps | grep 1` >/dev/console
 ps | grep 1 > /tmp/PS.txt
 sleep 1
 exec /bin/sh >/dev/console 2>&1
fi

A: exec switch does not work anymore to continue booting
B: kill[all] 1 [init] is a simple way to reboot immediately