Running with compressed kernelmodules

Under development: PCMCIA, wireless, etc.
Message
Author
User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

Running with compressed kernelmodules

#1 Post by Karl Godt »

As I am working on a post 30.5 kernel because the diff from 30.5 to 30.6 is pretty large (4000 lines) I started to have a look inside the initrd.gz. Using rcrsn51 script to extract the initrd I found that all kernel modules inside there are gzip compressed.

On my full install (Macpup 430/431) running a 30.9-i586-8cores-300Hz and a 30.9-i386-8cores-300Hz EXT3_DEFAULTS_TO_ORDERED=y kernel for testing which is running well so far (want to explore 1000Hz and -AMD architectures also)

I created a script to compress the modules at /lib/modules/`uname -r`/kernel which reduced the size of the modules from 48 MB to 20 MB :

Code: Select all

#!/bin/bash

case $1 in

-ko-gzip-func)

echo starting now gzip kernel modules script
cd /lib/modules/`uname -r`/kernel
PWD=`pwd`
echo $PWD
echo will start to transform ".ko" to ".ko.gz" in 10 sec
sleep 10
echo
echo 5th depth
 for i in */*/*/*/*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`;echo "$j"; [ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 5th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo 4th depth
 for i in */*/*/*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`;echo "$j"; [ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 4th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo 3th depth
 for i in */*/*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`;echo "$j"; [ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 3th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo 2th depth
 for i in */*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`; echo "$j";[ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 2th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo now running depmod so modprobe and modinfo find the modules again
depmod
echo finished running \(busybox\) depmod
echo
echo controlling it now
CPM=`cat /proc/modules | grep aes_generic`
echo $CPM
[ "$CPM" = "" ] && echo " probing always loading module aes_generic" && modinfo aes_generic && echo && modprobe aes_generic
echo tried to insert aes_generic
echo sleeping now 10 sec
sleep 10
echo removing aes_generic again
rmmod aes_generic
echo aes_generic removed
echo any warning or error messages occured \?
echo script finished
echo press ctrl and c to close the terminal window
while [ 2=2 ]; do sleep 4; done
exit 0
;;
*)
rxvt -e KO-GZIP.sh -ko-gzip-func
;;
esac
exit
I would like to find out if there is anybody who had positive or negative experiences using compressed kernel modules

and also if there is a more elegant way to find the .ko files and transform them to .ko.gz .

I could have posted this at the programming section also but I thought that the possibility to reduce the size of the pup-.sfs would fit here better.

PANZERKOPF
Posts: 282
Joined: Wed 16 Dec 2009, 21:38
Location: Earth

Re: Running with compressed kernelmodules

#2 Post by PANZERKOPF »

Karl Godt wrote: I would like to find out if there is anybody who had positive or negative experiences using compressed kernel modules
I had no problems with compressed modules. Note - modprobe must be compiled
with "--enable-zlib" for working with compressed modules. Busybox modprobe
also supports gzipped modules.
Karl Godt wrote: and also if there is a more elegant way to find the .ko files and transform them to .ko.gz .
cd /lib/modules/`uname -r`
rm modules.*
gzip -r *
depmod -a
Karl Godt wrote: I could have posted this at the programming section also but I thought that the possibility to reduce the size of the pup-.sfs would fit here better.
[/quote]
I think that is not correct because double compression always increases a size of the target file.
SUUM CUIQUE.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#3 Post by Karl Godt »

I think that is not correct because double compression always increases a size of the target file
It seems you are right :
mksquashfs4 :
22 MB of .ko.gz kernelmodules became 20 MB

48 Mb of .ko kernelmodules became 19 MB

( btw .: mksquashfs4 on the gzipped part was much faster than at the normal part )

The first boot was fine as usual but according to bootkernel.log took 5 seconds longer ( 26 against 21 sec. ) and according to messages 2 seconds ( 31.6 against 29.9 ) .

So I think I will experiment with uncompressed .ko inside the initrd.gz also .

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#4 Post by Karl Godt »

Seems that modprobe doesn't want to work on bzip2 modules
but
busybox modprobe does :?

PANZERKOPF
Posts: 282
Joined: Wed 16 Dec 2009, 21:38
Location: Earth

#5 Post by PANZERKOPF »

Karl Godt wrote: The first boot was fine as usual but according to bootkernel.log took 5 seconds longer ( 26 against 21 sec. ) and according to messages 2 seconds ( 31.6 against 29.9 ) .
Right, because modprobe needs additional time to uncompress each module.
Karl Godt wrote: Seems that modprobe doesn't want to work on bzip2 modules
but
busybox modprobe does
Never thought that busybox can do what normal util can't :)
Really Swiss army knife :)
SUUM CUIQUE.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#6 Post by Karl Godt »

Same with lzma -z ( -7 is the default )

48 MB > 17 MB

modprobe (-FULL) 3.6 refuses to load
modinfo 3.6 shows nothing
insmod 3.6 refuses to load

busybox 1.15.0 modprobe and insmod are working
but bb hasn't got the modinfo applet

mksquashfs4 :

17 MB > 17 MB : this could become an option to look further into .

Downloaded the module-init-tools 3.12 source from Debian : no lzma decompression but I uncommented the "Deprecated config file" warning at modprobe.c and depmod.c which is confusing quite some people at the lupu series .
I think I will test them for one week and upload them if they work without problems.

PANZERKOPF
Posts: 282
Joined: Wed 16 Dec 2009, 21:38
Location: Earth

#7 Post by PANZERKOPF »

Karl Godt wrote:Same with lzma -z ( -7 is the default )
modprobe (-FULL) 3.6 refuses to load
modinfo 3.6 shows nothing
insmod 3.6 refuses to load
busybox 1.15.0 modprobe and insmod are working
but bb hasn't got the modinfo applet
busybox has modinfo applet since 1.17 version. I compiled latest busybox and discovered some new applets (and bugfixes). I recommend you to build it.
SUUM CUIQUE.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#8 Post by Karl Godt »

busybox has modinfo applet since 1.17 version. I compiled latest busybox and discovered some new applets (and bugfixes). I recommend you to build it.
I already did it in autum (bb-1.17.1) and it took me several compils to detect that make menuconfig somehow disabled ash in the background and I wondered why puppy suddenly booted to desktop without loading any additinal modules. :shock: :oops:
So I added many features and busybox became between 750 and 1300 Kb and I was not able to disable hush . Some of the puppy scripts use #!/bin/ash like /etc/rc.d/rc.sysinit which is loading drivers in frugal installs for example.
And somewhere I think I saw a 1.17.2 already at wary094 or 098 .

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#9 Post by Karl Godt »

Ooops 1.18.2 out already : Seems they have fixed the FIX_ALIASES : no warnings at all at gcc 4.2.2 ,
Make menuconfig again disabled ash so I ran make defconfig and enabled just the full modutils and tune2fs : 728 Kb incl. vi .

The menuconfig is as large as at the kernel :? A good DOTconfig is always worth a lot .....

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#10 Post by technosaurus »

looking through the busybox source code indicates that it will even accept .xz modules - which should yield ~30% better compression and comparable decompression when compared to gzip

... and someone recently posted a couple helper scripts to the mailing list to make busybox mdev and modprobe work like full udev and full modprobe

http://lists.busybox.net/pipermail/busy ... 74344.html
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#11 Post by Karl Godt »

# /bin/BUSYBOX/busybox-1.18.2-ash-728Kb mdev --help
BusyBox v1.18.2 (2011-01-17 02:13:30 CET) multi-call binary.

Usage: mdev [-s]

-s Scan /sys and populate /dev during system boot

It can be run by kernel as a hotplug helper. To activate it:
echo /sbin/mdev > /proc/sys/kernel/hotplug
It uses /etc/mdev.conf with lines
[-]DEVNAME UID:GID PERM [>|=PATH] [@|$|*PROG]
seems to be a nice feature - didn't knew it before

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#12 Post by technosaurus »

I wrote a step by step howto for porting an applet to busybox here:
http://www.murga-linux.com/puppy/viewtopic.php?t=59776

I have the following applets in various stages of porting completion
  • autologinroot
    cddetect
    fusermount
    guess_fstype
    hotplug2stdout_notimeout
    minimp3
    oneit
    waitmax
    puppyinputdetect
    minimp3
    old-e2fstools
Many of these are part of the initrd and would thus benefit by being in busybox, if anyone is interested I can post the sources

minimp3 was just a hmm why not be able to have a bootup sound when sound is working, but then I came up with these 2 functions:

Code: Select all

play_au() { #just sends $@ to /dev/audio
cat $@ > /dev/audio
}

play_wav() { #just sends $@ to /dev/dsp
cat $@ > /dev/dsp
}
so when sound is setup you can just
cat /usr/share/audio/2barks.au >/dev/audio
or
cat /usr/share/audio/join.wav >/dev/dsp
(2barks.au is <5kb, join.wav is <4kb and both use existing applets)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#13 Post by amigo »

cd /lib/modules/`uname -r`/kernel
PWD=`pwd`

# replace everything in your script down to the line 'depmod' with this:
find -type f -name "*.ko" -exec gzip -9 {} \;

PANZERKOPF
Posts: 282
Joined: Wed 16 Dec 2009, 21:38
Location: Earth

#14 Post by PANZERKOPF »

technosaurus wrote: ... and someone recently posted a couple helper scripts to the mailing list to make busybox mdev and modprobe work like full udev and full modprobe
http://lists.busybox.net/pipermail/busy ... 74344.html
Thanks for information. I knew about mdev but latest versions are improved (e.g. firmware loading). It hasn't full udev functionality but can be useful for simple systems based on Linux.
SUUM CUIQUE.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#15 Post by Karl Godt »

find -type f -name "*.gz" -exec basename {} \;
Superb solution !

This is part of find --help :
-exec COMMAND ; -exec COMMAND {} + -ok COMMAND ;
but on both rxvt and sakura the backslash is needed !

Just because of couriousity I tried to find a replacement for the {} but gave up :

Code: Select all

TOTALnumberofMODULES=`ls -1R |grep ".gz" | wc -l`
ls -R | grep ".gz" > LS.txt
while [ A != $TOTALnumberofMODULES ]; do
(( c++ ))
module$c=`sed -n "$c"p LS.txt` # this doesn't work
echo module$c
done
Really something useful these {} !

I also compiled 6 times busybox to get it down to the size of 1.15.0
with cp the output of busybox to geany , saving it as BB-1.17.1-419kb-functions.txt file and doing some

Code: Select all

cat /root/BB-1.17.1-419kb-functions.txt | tr "," "\n" | tr -d "\t" | tr -d " " > BB-1.17.1-419-2
diff /root/BB-1.15.0-2 /root/BB-1.17.1-419-2
comparisons.
Afterwards I mounted a 511 which has got a busybox 1.16.2 of 624kb ( 1.15.0 is 427kb ) that has got a lot more compression features and internet functions enabled . Though some functions like date or eject are enabled there, the full binaries still remain in the path .
The most unexpected had been, that the 511 BB 1.16.2 is glibc2.6.1 compatible and is running here at 430/1 .

And about hotplugging daemons : If they create device nodes but also delete them after unplugging , could cause some confusions if /dev/sdb isn't available the next day.

For anyone who is interested :
I attach my basic BB .config : The functions are the same as at the puppy-bb-1.15.0 with only additionally lsusb enabled which I had been really missing. I hope the sub-functions are ok, too.
Attachments
DOTconfig-BB-1.17.1-419kb.gz
(5.94 KiB) Downloaded 521 times

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#16 Post by Karl Godt »

Now I had the time to apply

Code: Select all

find . | cpio -o -H newc | gzip -9 > ../initrd-ko-modules.gz
from rcrsn51 inside
pwd
/root/my-applications/initrd-FOX-3-NEW
after I ran

Code: Select all

find -type f -name "*.gz" -exec gunzip {} \;
inside
pwd
/root/my-applications/initrd-FOX-3-NEW/lib
and the second time
I think that is not correct because double compression always increases a size of the target file.
is true :
original initrd.gz with .ko.gz inside : 1394 K
new initrd.gz with .ko inside : 1368 K

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#17 Post by technosaurus »

goingnuts did some experiments in pupngo and IIRC when he built the sfs file (or was it the initrd?) with gzipped modules, it actually got bigger - wondering if xz would fare similarly, I'll add it to the todo list for features in the zdrv cutter (defaulting to gz, but with option for others)

the lzma/xz maintainer (lasse collin) posted kernel and squashfs patches too
http://www.mail-archive.com/linux-embed ... ernel.org/
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#18 Post by amigo »

Double-compression should always make things bigger, since each file must have the decompressor code added to it. xz is considerably slower than gzip, though for single small items it shouldn't make too much difference. I am surprised at the amount of compression you guys are reporting regardless of which tool is being used -most kernel code(modules) is quite dense and doesn't compress much no matter what you use. Another consideration is that when you use compressed items it takes RAM space to decompress them. This can become problematic when the items are large -especially with lzma/xz as it takes much more RAM than other compression methods.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#19 Post by Karl Godt »

Another consideration is that when you use compressed items it takes RAM space to decompress them. This can become problematic when the items are large -especially with lzma/xz as it takes much more RAM than other compression methods.
The first impressions are that way but it depends on the level . What I would need are approx. numbers like twice as much or 1.2 as much and this kind .


I am running now my first own modified initrd.gz at a frugall installation which worked both with and without loading the savefile . Adding a lot of unarchival utils and module-tools and fsck got me a 748 kb bb-1.17.1 and another with additional tools for the rdsh like less,more,ed,blkid,uptime,which,diff,... made the bb growing to 832kb. Including the original e2fsck which is 173kb but with uncompressed kernelmodules the initrd.gz became 1383kb instead of the original 1392kb. The first two boots paniced because I had forgotten to adjust the entries at /lib/modules/2.6.30.5/modules.dep from *.ko.gz to *.ko inside the initrd folder . If lzma *.ko would reduce the size of the modules which are uncompressed 1003kb and gzip 407kb down to 300-350kb I would be happy because this would enable some console and init features at bb .

And puppy for older hardware ? There are actually no kernels for real old hardware anymore . The original BB of the initrd.gz is version 1.4.2 with a size of 253kb statically . The first 1.17.1 that resembled the 1.4.2 I made several ours ago had been 728kb statically.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#20 Post by Karl Godt »

One disadvantage of the BB find applet is the missing "-noleaf" option that is used to
"\-noleaf"
Do not optimize by assuming that directories contain 2 fewer
subdirectories than their hard link count. This option is needed when
searching filesystems that do not follow the Unix directory-link
convention, such as CD-ROM or MS-DOS filesystems or AFS volume mount
points. Each directory on a normal Unix filesystem has at least 2
hard links: its name and its `.' entry. Additionally, its
subdirectories (if any) each have a `..' entry linked to that
directory. When
find
is examining a directory, after it has statted 2 fewer subdirectories
than the directory's link count, it knows that the rest of the entries
in the directory are non-directories (`leaf' files in the directory
tree). If only the files' names need to be examined, there is no need
to stat them; this gives a significant increase in search speed.
two times at the init[.sh] while version upgrade :

Code: Select all

if [ $NEWPVERSION -gt $OLDPVERSION ];then
  echo -e "\\033[1;35m"  >/dev/console #35=purple.
  echo "Version update, restoring 'official' files, please wait..." >/dev/console
  echo -en "\\033[0;39m" >/dev/console

#v2.12 this while-loop has become ultra-slow... seems upx compressed execs
  #responsible, recently upx-ed all the execs, now restore busybox, grep, cp.
  DOTCNT=0
  find ./ -noleaf -type f | sed -e 's/^\.//g' |
and :

Code: Select all

#need to cleanup whiteout files if a new .sfs layer has been added...
 if [ "$NEWUNIONRECORD" != "" -o "$PPURGE" = "yes" ];then
  #find all .wh.__dir_opaque files at the OLDFILESMNTPT layer... v424 bugfixes...
  cd $OLDFILESMNTPT
  find ./ -noleaf -type f -name ".wh.*" | sed -e 's/^\.//g' |
I don"t know if this does matter or if there are option combinations that would achieve the same. I am also not able to compile findutils ( 4.1.20 , 4.4.0 and 4.5 9 ) statically , even after googling around one or two hours. :x
The configure option description is very common and not very specific :
--disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no)
--enable-FEATURE[=ARG] include FEATURE [ARG=yes]
--enable-leaf-optimisation
Enable an optimisation which saves lstat calls to
identify subdirectories on filesystems having
traditional Unix semantics
--enable-d_type-optimization
Make use of the file type data returned in struct
dirent.d_type by readdir()
--enable-d_type-optimisation
Synonym for --enable-d_type-optimization

Post Reply