Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 16 Oct 2019, 06:04
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » System
Remove unneeded modules and firmware - Zdrv_Cutter
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [15 Posts]  
Author Message
technosaurus


Joined: 18 May 2008
Posts: 4869
Location: Blue Springs, MO

PostPosted: Tue 19 Jan 2010, 23:39    Post subject:  Remove unneeded modules and firmware - Zdrv_Cutter
Subject description: cutdown over 50MB of extra bloat (~20MB compressed)
 

Thanks to amigo for the assist on this one...

This should remove about 20MB compressed of unneeded modules from your zdrv.... or during a remaster you can delete the contents of /lib/modules from your pupXXXsfs working directory and place the zdrv file in the iso directory ... thats about 1/5th the size of the iso

This is something I thought might be useful for the hardware specific distros to (pupeee, olpc etc...) and for low resource computers that still want to do a frugal install... or just the minimalist at heart

0.1 - just cutdown modules - still add all firmware(10-15MB)
0.2 - cutdown both modules and firmware(~5MB)
0.3 - removes initrd modules (~1MB)
TODO
#0.4 cover /lib/firmware (~2MB)
#0.5 cover xorg drivers (~4MB)
#0.6 patch woof to allow xorg drivers in zdrv
#0.7 patch remaster script to integrate this as option
#0.8 autobuild using /etc/modules/firmware.dep{,(uname -r)}
#0.9 bugfixes and GUI improvements
#1.0 Final release

#1.X main sfs support: stripping binaries (strip), removing manpages locales, DEV files and broken links; optimize images (optipng), symlink duplicate files (rdfind), compress large binaries (upx)

Development thread here:
http://www.murga-linux.com/puppy/viewtopic.php?t=51450
zdrv_cutter-0.3.pet
Description  also removes initrd modules since they get built automatically
adds some dialogs and chance to edit/check the build
pet

 Download 
Filename  zdrv_cutter-0.3.pet 
Filesize  2.13 KB 
Downloaded  1571 Time(s) 
zdrv_cutter-0.2.pet
Description  cuts both modules and firmware
pet

 Download 
Filename  zdrv_cutter-0.2.pet 
Filesize  2.02 KB 
Downloaded  1328 Time(s) 
zdrvcutr.gz
Description  original script - just for reference
gz

 Download 
Filename  zdrvcutr.gz 
Filesize  825 Bytes 
Downloaded  1328 Time(s) 

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.

Last edited by technosaurus on Thu 21 Jan 2010, 10:25; edited 5 times in total
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 4869
Location: Blue Springs, MO

PostPosted: Wed 20 Jan 2010, 00:13    Post subject:  

firmware is in tar.gz thus the loaded modules don't necessarily reference them, so the will need to be copied manually for now

example:
iwlwifi contains iwlwifi-3945... but the module name is iwl3945

iwlcore should get /lib/modules/all-firmware/iwlwifi.tar.gz

I added a couple lines that copy them all until we can figure out the cases (still cuts out 15MB)-

Does someone have a list of what module gets what firmware? rerwin? tempestuous?
Code:
#!/bin/sh
. /etc/DISTRO_SPECS #gets $DISTRO* variables
KERNELVER=$(uname -r) # get the version of your kernel
PREFIX1CHAR="`echo -n "$DISTRO_FILE_PREFIX" | cut -c 1`"
KERNEL3CHARS="`echo -n "$KERNELVER" | tr -d '.' | tr -d '\-' | tr -d '[a-z]' | rev | cut -c 1,2,3 | rev`"
ZDIR="z${PREFIX1CHAR}${DISTRO_VERSION}${KERNEL3CHARS}"
# set this to the name of a temp dir:
TMPDIR=/dev/shm/$ZDIR #faster to use /dev/shm
# create the module directory in the TMP dir:
mkdir -p $TMPDIR/all-firmware
lsmod |tail +2 |cut -f1 -d' ' > $TMPDIR/mods #just a list of modules
sed -i "s/_/?/g" $TMPDIR/mods #replaces the _ with ? so that - and . will not give false evaluation
cd /
for module in $(cat $TMPDIR/mods) ; do
   # uncomment the first and comment the second one to just see the list
   # find lib/modules/$KVERS -type f -name "$module*"
   find lib/modules/$KVERS -type f -name "$module*" -exec cp --parents {} $TMPDIR/ \;
done
cp /lib/modules/all-firmware/* $TMPDIR/lib/modules/all-firmware #needed until we make case for each tarball
mv /lib/modules /lib/modules~ # can't force a link over a directory
ln -s -f $TMPDIR/lib/modules /lib #use link so busybox depmod works
depmod #-b $TMPDIR/lib/modules/$KERNELVER -e -F /path/to/kernel/System.map
rm $TMPDIR/mods #cleanup
dir2sfs $TMPDIR
mv $TMPDIR"_"$DISTRO_VERSION.sfs ~/$ZDIR.sfs #move to $HOME
mv $TMPDIR"_"$DISTRO_VERSION.sfs-md5.txt ~/$ZDIR.sfs-md5.txt #move to $HOME
rm -rf $TMPDIR
rm -rf /lib/modules
mv /lib/modules~ /lib/modules
Xdialog --left --buttons-style text --title "Zdrv Cutter" --ok-label "Show file" --cancel-label "Exit" --yesno "Cutdown z"${PREFIX1CHAR}${DISTRO_VERSION}${KERNEL3CHARS}".sfs is now in ~" 0 0
[ $? -eq 0 ] && rox ~ &

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 4869
Location: Blue Springs, MO

PostPosted: Wed 20 Jan 2010, 21:26    Post subject:  

beta 0.2 is now released (see initial post)

added firmware cutdown section using info from /etc/modules/firmware.dep - removes an additional 5MB compressed

a typical zdrv is now ~4.5 MB instead of 25 MB

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 4869
Location: Blue Springs, MO

PostPosted: Wed 20 Jan 2010, 22:23    Post subject:  

beta 0.3 is now released

removes /lib/modules/$(uname -r)/initrd since it gets built at startup
added some dialogs giving opportunity to check before and after it is built
fixed some typos
zdrv_cutter-0.3.pet
Description 
pet

 Download 
Filename  zdrv_cutter-0.3.pet 
Filesize  2.13 KB 
Downloaded  1310 Time(s) 

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 4869
Location: Blue Springs, MO

PostPosted: Thu 21 Jan 2010, 01:05    Post subject:  

you can do something similar with xorg drivers ~4MB

#this code tells you your xorg driver
grep -w card0driver /etc/X11/xorg.conf |cut -d "\"" -f 2

#files are located in /usr/X11R7/lib/xorg/modules/drivers/
#however some end in _drv.so and some just in .so
#but the ones without _drv are insignificantly small

There is no reason that these cannot go in the zdrv either AFAIK

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
greengeek


Joined: 20 Jul 2010
Posts: 5623
Location: Republic of Novo Zelande

PostPosted: Wed 13 Feb 2013, 14:25    Post subject:  

Does a smaller zdrv bring the benefit of a smaller RAM footprint and speed gains?
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4869
Location: Blue Springs, MO

PostPosted: Wed 13 Feb 2013, 14:46    Post subject:  

That depends on where the modules are originally located:
if they are in the pup*.sfs, you would need to remaster and it would reduce union/mount/decompression overhead and more with pfix=ram.
if they are already in a separate, larger zdrv.sfs, then it reduces the union/mount/decompression overhead.

Side note: after learning how modules are loaded (the entire module is memmapped first and then gets passed to init_module), I would guess that its possible to load the modules and then completely umount the zdrv.sfs

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
moeppyfan


Joined: 11 Feb 2016
Posts: 48

PostPosted: Fri 19 Feb 2016, 12:12    Post subject:  

Hello technosaurus,

I tried your zdrv_cutter-0.3.pet on huge-kernel-3.14.55 (Slacko 6) and it shrinks my zdrv from 30MB to 2.8MB! Nice. The new 2.8MB zdrv boots to desktop - and trackpad, keyboard, video, sound all work on my Dell Latitude D410.

To get it to recognize my wireless, I had to recreate /lib/firmware dir and re-add these files:

Code:
/lib/firmware/ipw2200-bss.fw
/lib/firmware/ipw2200-ibss.fw
/lib/firmware/ipw2200-sniffer.fw

And to get "depmod" to run without error, I had to re-add these two files:

Code:
/lib/modules/3.14.55/modules.builtin
/lib/modules/3.14.55/modules.order

I'm not sure if copying that old module list is 100% accurate though, since stuff has been cut?
Everything seems to work now, so I'll keep using your cut version, and let you know if I spot anything else. Smile

P.S. The user backi started a thread about your script here, which is how I found out about it.

edit: Tried also on Dell Latitude E4300. The process was similar, except I had to re-add /lib/firmware/b43 folder for wireless.
Back to top
View user's profile Send private message 
Pelo

Joined: 10 Sep 2011
Posts: 12591
Location: Mer méditerrannée (1 kms°)

PostPosted: Sat 26 Mar 2016, 15:28    Post subject: I will try it with iwlwifi.  

Very usefull topic, 2016 most of us surf the web with at laptop at home. Wireless just needed for your firmware sure makes the Puppy Lighter.
I will try it with iwlwifi.
zdrv_vivid_6.5.sfs: 30 M
staging: 3735 K seems a place to study.. but all that is in english, and i am not a programer.

_________________
Passenger Pelo ! don't ask him to repair the aircraft. Don't use him as a demining dog .... pleeease.
Back to top
View user's profile Send private message Yahoo Messenger 
jabu2

Joined: 07 Apr 2008
Posts: 46
Location: Australia

PostPosted: Sat 26 Mar 2016, 23:06    Post subject:    

Likewise not a programmer,
but TechnoS has once more come up with what looks like something really useful -if I'm understanding his concepts.

TechnoS identified it as "for low resource computers that still want to do a frugal install... or just the minimalist at heart ".

I think its also what is needed for a custom job, to use on a laptop/portable taken travelling - running on CD or USB.
Minimalist, but secure, and low risk of loss (if running on an oldie but goodie), uncluttered but doing the basics needed: =email, internet (top browser), file managing for whatever storage/dumping space the oldie has, etc, and easy wifi and internet connection?

Is that correct? Or are there better options for such a travelling OS?

Thanks TechnoSaurus for your efforts/initiative,
and to anyone who can give us an end-product using Google chrome and simple connectivity choices built-in.

ATVB (all the very best)
jabu2
Back to top
View user's profile Send private message 
stemsee

Joined: 27 Jun 2013
Posts: 2483
Location: In The Way

PostPosted: Mon 02 May 2016, 06:18    Post subject:  

When will 1.0 be available?
Back to top
View user's profile Send private message MSN Messenger 
Sailor Enceladus

Joined: 22 Feb 2016
Posts: 1565

PostPosted: Sat 03 Sep 2016, 20:26    Post subject:  

stemsee wrote:
When will 1.0 be available?

I would love a version 0.3b that patches modules.builtin and modules.order to match the new zdrv, rather than deletes them.
Back to top
View user's profile Send private message 
Pelo

Joined: 10 Sep 2011
Posts: 12591
Location: Mer méditerrannée (1 kms°)

PostPosted: Thu 03 Aug 2017, 14:41    Post subject: being tested with my X-Slim Slacko  

being tested with my X-Slim Slacko backi information;
however i must keep iwlwifi, which is one of the biggest so no many MBs to delete

_________________
Passenger Pelo ! don't ask him to repair the aircraft. Don't use him as a demining dog .... pleeease.
Back to top
View user's profile Send private message Yahoo Messenger 
technosaurus


Joined: 18 May 2008
Posts: 4869
Location: Blue Springs, MO

PostPosted: Sun 13 Aug 2017, 19:14    Post subject:  

I don't really use this method now since the kernel's introduction of allmodconfig and allyesconfig.
make allmodconfig rebuilds the kernel without support for unused modules.
make allyesconfig will then build those used modules into the kernel
(for the full effect you need to boot into the former before building the latter)

The down side is that you need the devx and kernel source (I normally do), so I can see how this may still be useful.

most of the modules.* files are either completely unnecessary or automatically rebuilt by other things like depmod and will end up just getting duplicated in the savefile, so putting it in the sfs would be wasteful and redundant.

I may be able to do something about keeping the appropriate firmware though.

I have to stress that this method is just a hack. If you are doing a commercial product, the above kernel build method is more appropriate (probably followed by iterations of make menuconfig for anticipated upgrades)

_________________
Check out my github repositories. I may eventually get around to updating my blogspot.
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 3539

PostPosted: Mon 22 Jul 2019, 07:20    Post subject:  

technosaurus wrote:
I don't really use this method now since the kernel's introduction of allmodconfig and allyesconfig.
make allmodconfig rebuilds the kernel without support for unused modules.
make allyesconfig will then build those used modules into the kernel
(for the full effect you need to boot into the former before building the latter)

The down side is that you need the devx and kernel source (I normally do), so I can see how this may still be useful.

Fumbling my way through building a kernel, using fatdog 8.2 I downloaded the devx and kernel sources sfs and (booted frugally from a usb) unsquashed the kernel source sfs on my hdd and sfs loaded the devx.

Ran 'make' and it took hours. I even ctrl-c broke out of that as it had passed my zzzz time. The next day (or rather later that same day) I and ran make localyesconfig and then ran make again. It seemingly picked up from where it had left off. A hour or so later and it reported it had created the bzImage.

With the bzImage built I renamed that vmlinuz-mine. Made a folder /boot and copied that into there, along with the System.map. With my frugal booted normally, usb left attached, wireless net connected, I then ran make modules install. Don't think that was necessary as it just created another vmlinuz the same size as the first one.

Opened the default fatdog initrd and removed fd64.sfs (main sfs) and kernel-modules.sfs out of that. Reclosed it and gzip'd that initrd (that I named initrd-mine.gz).

Around 16MB combined vmlinuz and initrd.gz ... that boots fine. Within init (/sbin/system-init in fatdogs case) I exit to shell prior to the fd64.sfs loading (aufs layering) being set up, so this is just a tui/cli based boot. But it boots with wireless net connected OK and reads the usb ok. Which is all I want for this particular boot i.e. I use that busybox/initrd based boot to ssh into hashbang - from where I have tmux (multiple windows), irc, email, w3m web browser, mail lists, BBS access ...etc. (more than enough to keep me occupied when otherwise bored). As expected from such a small size, it boots very quickly even from usb. No pseudo terminals, no X, so very secure. And as I ssh into hashbang to run things from there my ISP/state only sees that encrypted link, no dns or other traffic. And remote sites only see the hashbang servers IP, not my IP. The likes of reddit (ssh redditbox.us) and yahoo finance using w3m are lightning fast, whereas when accessed using chrome they're more often relatively slow/laggy. Booted (from usb) and then the usb disconnected (running in ram), it uses around 76MB of ram. I have a swap partition on HDD and within init I set that up as a encrypted swap, so even though running in ram (4GB) it has that swap space (16GB partition) on top of that. To test that out I dd'd a 10GB file and that was created fine and the system continued to function as normal.

But as the attached image indicates (blurry text due to scaling down image - a clearer version is available here), you do need to recompile things again if new hardware is going to be used/attached. With the kernel built once however subsequent compiles run through much much quicker as it only recompiles the things that have been changed.
localyesconfig400.png
 Description   
 Filesize   72.77 KB
 Viewed   134 Time(s)

localyesconfig400.png


_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [15 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » System
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0825s ][ Queries: 12 (0.0212s) ][ GZIP on ]