Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 21 May 2013, 04:36
All times are UTC - 4

 Forum index » Advanced Topics » Cutting edge
Running with compressed kernelmodules
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 4 [47 Posts]   Goto page: 1, 2, 3, 4 Next
Author Message
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Sun 16 Jan 2011, 13:35    Post subject:  Running with compressed kernelmodules  
As I am working on a post 30.5 kernel because the diff from 30.5 to 30.6 is pretty large (4000 lines) I started to have a look inside the initrd.gz. Using rcrsn51 script to extract the initrd I found that all kernel modules inside there are gzip compressed.

On my full install (Macpup 430/431) running a 30.9-i586-8cores-300Hz and a 30.9-i386-8cores-300Hz EXT3_DEFAULTS_TO_ORDERED=y kernel for testing which is running well so far (want to explore 1000Hz and -AMD architectures also)

I created a script to compress the modules at /lib/modules/`uname -r`/kernel which reduced the size of the modules from 48 MB to 20 MB :
Code:
#!/bin/bash

case $1 in

-ko-gzip-func)

echo starting now gzip kernel modules script
cd /lib/modules/`uname -r`/kernel
PWD=`pwd`
echo $PWD
echo will start to transform ".ko" to ".ko.gz" in 10 sec
sleep 10
echo
echo 5th depth
 for i in */*/*/*/*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`;echo "$j"; [ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 5th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo 4th depth
 for i in */*/*/*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`;echo "$j"; [ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 4th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo 3th depth
 for i in */*/*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`;echo "$j"; [ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 3th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo 2th depth
 for i in */*; do echo "$i" | grep '.ko'; j=`echo "$i" | grep '.ko'`; echo "$j";[ ! "$j" = "" ] && k=`realpath "$j"`&& echo "$k" && gzip "$k";echo; done
echo finished to gzip 2th depth kernelmodules
echo sleeping now 10 sec
sleep 10
echo
echo now running depmod so modprobe and modinfo find the modules again
depmod
echo finished running \(busybox\) depmod
echo
echo controlling it now
CPM=`cat /proc/modules | grep aes_generic`
echo $CPM
[ "$CPM" = "" ] && echo " probing always loading module aes_generic" && modinfo aes_generic && echo && modprobe aes_generic
echo tried to insert aes_generic
echo sleeping now 10 sec
sleep 10
echo removing aes_generic again
rmmod aes_generic
echo aes_generic removed
echo any warning or error messages occured \?
echo script finished
echo press ctrl and c to close the terminal window
while [ 2=2 ]; do sleep 4; done
exit 0
;;
*)
rxvt -e KO-GZIP.sh -ko-gzip-func
;;
esac
exit

I would like to find out if there is anybody who had positive or negative experiences using compressed kernel modules

and also if there is a more elegant way to find the .ko files and transform them to .ko.gz .

I could have posted this at the programming section also but I thought that the possibility to reduce the size of the pup-.sfs would fit here better.
Back to top
View user's profile Send private message Visit poster's website 
PANZERKOPF

Joined: 16 Dec 2009
Posts: 249
Location: Earth
PostPosted: Sun 16 Jan 2011, 14:38    Post subject: Re: Running with compressed kernelmodules  
Karl Godt wrote:

I would like to find out if there is anybody who had positive or negative experiences using compressed kernel modules

I had no problems with compressed modules. Note - modprobe must be compiled
with "--enable-zlib" for working with compressed modules. Busybox modprobe
also supports gzipped modules.
Karl Godt wrote:

and also if there is a more elegant way to find the .ko files and transform them to .ko.gz .

cd /lib/modules/`uname -r`
rm modules.*
gzip -r *
depmod -a
Karl Godt wrote:

I could have posted this at the programming section also but I thought that the possibility to reduce the size of the pup-.sfs would fit here better.

[/quote]
I think that is not correct because double compression always increases a size of the target file.
_________________
SUUM CUIQUE.
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Sun 16 Jan 2011, 15:45    Post subject:  
Quote:
I think that is not correct because double compression always increases a size of the target file

It seems you are right :
mksquashfs4 :
22 MB of .ko.gz kernelmodules became 20 MB

48 Mb of .ko kernelmodules became 19 MB

( btw .: mksquashfs4 on the gzipped part was much faster than at the normal part )

The first boot was fine as usual but according to bootkernel.log took 5 seconds longer ( 26 against 21 sec. ) and according to messages 2 seconds ( 31.6 against 29.9 ) .

So I think I will experiment with uncompressed .ko inside the initrd.gz also .
Back to top
View user's profile Send private message Visit poster's website 
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Sun 16 Jan 2011, 16:40    Post subject:  
Seems that modprobe doesn't want to work on bzip2 modules
but
busybox modprobe does Confused
Back to top
View user's profile Send private message Visit poster's website 
PANZERKOPF

Joined: 16 Dec 2009
Posts: 249
Location: Earth
PostPosted: Sun 16 Jan 2011, 18:16    Post subject:  
Karl Godt wrote:

The first boot was fine as usual but according to bootkernel.log took 5 seconds longer ( 26 against 21 sec. ) and according to messages 2 seconds ( 31.6 against 29.9 ) .

Right, because modprobe needs additional time to uncompress each module.

Karl Godt wrote:

Seems that modprobe doesn't want to work on bzip2 modules
but
busybox modprobe does

Never thought that busybox can do what normal util can't Smile
Really Swiss army knife Smile
_________________
SUUM CUIQUE.
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Sun 16 Jan 2011, 19:24    Post subject:  
Same with lzma -z ( -7 is the default )

48 MB > 17 MB

modprobe (-FULL) 3.6 refuses to load
modinfo 3.6 shows nothing
insmod 3.6 refuses to load

busybox 1.15.0 modprobe and insmod are working
but bb hasn't got the modinfo applet

mksquashfs4 :

17 MB > 17 MB : this could become an option to look further into .

Downloaded the module-init-tools 3.12 source from Debian : no lzma decompression but I uncommented the "Deprecated config file" warning at modprobe.c and depmod.c which is confusing quite some people at the lupu series .
I think I will test them for one week and upload them if they work without problems.
Back to top
View user's profile Send private message Visit poster's website 
PANZERKOPF

Joined: 16 Dec 2009
Posts: 249
Location: Earth
PostPosted: Sun 16 Jan 2011, 20:08    Post subject:  
Karl Godt wrote:
Same with lzma -z ( -7 is the default )
modprobe (-FULL) 3.6 refuses to load
modinfo 3.6 shows nothing
insmod 3.6 refuses to load
busybox 1.15.0 modprobe and insmod are working
but bb hasn't got the modinfo applet

busybox has modinfo applet since 1.17 version. I compiled latest busybox and discovered some new applets (and bugfixes). I recommend you to build it.
_________________
SUUM CUIQUE.
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Sun 16 Jan 2011, 20:48    Post subject:  
Quote:
busybox has modinfo applet since 1.17 version. I compiled latest busybox and discovered some new applets (and bugfixes). I recommend you to build it.


I already did it in autum (bb-1.17.1) and it took me several compils to detect that make menuconfig somehow disabled ash in the background and I wondered why puppy suddenly booted to desktop without loading any additinal modules. Shocked Embarassed
So I added many features and busybox became between 750 and 1300 Kb and I was not able to disable hush . Some of the puppy scripts use #!/bin/ash like /etc/rc.d/rc.sysinit which is loading drivers in frugal installs for example.
And somewhere I think I saw a 1.17.2 already at wary094 or 098 .
Back to top
View user's profile Send private message Visit poster's website 
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Sun 16 Jan 2011, 21:46    Post subject:  
Ooops 1.18.2 out already : Seems they have fixed the FIX_ALIASES : no warnings at all at gcc 4.2.2 ,
Make menuconfig again disabled ash so I ran make defconfig and enabled just the full modutils and tune2fs : 728 Kb incl. vi .

The menuconfig is as large as at the kernel Confused A good DOTconfig is always worth a lot .....
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 3843
PostPosted: Sun 16 Jan 2011, 23:15    Post subject:  
looking through the busybox source code indicates that it will even accept .xz modules - which should yield ~30% better compression and comparable decompression when compared to gzip

... and someone recently posted a couple helper scripts to the mailing list to make busybox mdev and modprobe work like full udev and full modprobe

http://lists.busybox.net/pipermail/busybox/2011-January/074344.html
_________________
Puppy Web Desktop Now with pet packages - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Mon 17 Jan 2011, 00:02    Post subject:  
Quote:
# /bin/BUSYBOX/busybox-1.18.2-ash-728Kb mdev --help
BusyBox v1.18.2 (2011-01-17 02:13:30 CET) multi-call binary.

Usage: mdev [-s]

-s Scan /sys and populate /dev during system boot

It can be run by kernel as a hotplug helper. To activate it:
echo /sbin/mdev > /proc/sys/kernel/hotplug
It uses /etc/mdev.conf with lines
[-]DEVNAME UID:GID PERM [>|=PATH] [@|$|*PROG]

seems to be a nice feature - didn't knew it before
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 3843
PostPosted: Mon 17 Jan 2011, 00:55    Post subject:  
I wrote a step by step howto for porting an applet to busybox here:
http://www.murga-linux.com/puppy/viewtopic.php?t=59776

I have the following applets in various stages of porting completion
    autologinroot
    cddetect
    fusermount
    guess_fstype
    hotplug2stdout_notimeout
    minimp3
    oneit
    waitmax
    puppyinputdetect
    minimp3
    old-e2fstools

Many of these are part of the initrd and would thus benefit by being in busybox, if anyone is interested I can post the sources

minimp3 was just a hmm why not be able to have a bootup sound when sound is working, but then I came up with these 2 functions:

Code:
play_au() { #just sends $@ to /dev/audio
cat $@ > /dev/audio
}

play_wav() { #just sends $@ to /dev/dsp
cat $@ > /dev/dsp
}


so when sound is setup you can just
cat /usr/share/audio/2barks.au >/dev/audio
or
cat /usr/share/audio/join.wav >/dev/dsp
(2barks.au is <5kb, join.wav is <4kb and both use existing applets)
_________________
Puppy Web Desktop Now with pet packages - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 1757
PostPosted: Mon 17 Jan 2011, 10:17    Post subject:  
cd /lib/modules/`uname -r`/kernel
PWD=`pwd`

# replace everything in your script down to the line 'depmod' with this:
find -type f -name "*.ko" -exec gzip -9 {} \;
Back to top
View user's profile Send private message 
PANZERKOPF

Joined: 16 Dec 2009
Posts: 249
Location: Earth
PostPosted: Mon 17 Jan 2011, 11:21    Post subject:    
technosaurus wrote:

... and someone recently posted a couple helper scripts to the mailing list to make busybox mdev and modprobe work like full udev and full modprobe
http://lists.busybox.net/pipermail/busybox/2011-January/074344.html

Thanks for information. I knew about mdev but latest versions are improved (e.g. firmware loading). It hasn't full udev functionality but can be useful for simple systems based on Linux.
_________________
SUUM CUIQUE.
Back to top
View user's profile Send private message 
Karl Godt


Joined: 20 Jun 2010
Posts: 2675
Location: Kiel,Germany
PostPosted: Mon 17 Jan 2011, 18:46    Post subject:    
Quote:
find -type f -name "*.gz" -exec basename {} \;

Superb solution !

This is part of find --help :
Quote:
-exec COMMAND ; -exec COMMAND {} + -ok COMMAND ;

but on both rxvt and sakura the backslash is needed !

Just because of couriousity I tried to find a replacement for the {} but gave up :
Code:
TOTALnumberofMODULES=`ls -1R |grep ".gz" | wc -l`
ls -R | grep ".gz" > LS.txt
while [ A != $TOTALnumberofMODULES ]; do
(( c++ ))
module$c=`sed -n "$c"p LS.txt` # this doesn't work
echo module$c
done

Really something useful these {} !

I also compiled 6 times busybox to get it down to the size of 1.15.0
with cp the output of busybox to geany , saving it as BB-1.17.1-419kb-functions.txt file and doing some
Code:
cat /root/BB-1.17.1-419kb-functions.txt | tr "," "\n" | tr -d "\t" | tr -d " " > BB-1.17.1-419-2
diff /root/BB-1.15.0-2 /root/BB-1.17.1-419-2

comparisons.
Afterwards I mounted a 511 which has got a busybox 1.16.2 of 624kb ( 1.15.0 is 427kb ) that has got a lot more compression features and internet functions enabled . Though some functions like date or eject are enabled there, the full binaries still remain in the path .
The most unexpected had been, that the 511 BB 1.16.2 is glibc2.6.1 compatible and is running here at 430/1 .

And about hotplugging daemons : If they create device nodes but also delete them after unplugging , could cause some confusions if /dev/sdb isn't available the next day.

For anyone who is interested :
I attach my basic BB .config : The functions are the same as at the puppy-bb-1.15.0 with only additionally lsusb enabled which I had been really missing. I hope the sub-functions are ok, too.
DOTconfig-BB-1.17.1-419kb.gz
Description 
gz

 Download 
Filename  DOTconfig-BB-1.17.1-419kb.gz 
Filesize  5.94 KB 
Downloaded  247 Time(s) 
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 1 of 4 [47 Posts]   Goto page: 1, 2, 3, 4 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

[ Time: 0.0870s ][ Queries: 12 (0.0075s) ][ GZIP on ]