pUPnGO - 6Mb ISO - Basic Building Block Puplet

[technosaurus]

Run "initrd_bundle install" as first thing in init.
I know some of this should be included via BB but...

Reduced the size from 226K to 137K - oh yeah! But just to get an estimate of the potential. Its a 40% reduction and if the more heavy stuff like ntfs-3g and e2fsck could be included too its worth the huge amount of time used to make it work...
What to do with multiple defs if they reside in a local lib needed to make an app (ex. merging fileutils and findutils both need their own version of getop.c)?

you can either make them compatible by modifying one or both, or just rename one and the references to it

Re busybox apps
elspci -> lspci (with some script mods)
disktype (from guessfstype?) there was a patch for blkid that does the same thing, but it wasn't mainlined
find -> find (depending on if you can mod the script that needs it)
(i do have busybox patches for guessfstype and olde2fsprogs as well as minimp3)

[faifpuhp]

This is really awesome, I'm glad it's still around (first post from 2010?)

[technosaurus]

also if you become familiar with /proc and /sys a lot of tools can be replaced with scripts
this is my eject script

Code: Select all

#!/bin/sh
echo "1" > /proc/sys/dev/cdrom/autoeject
umount /dev/cdrom

uname -a == read A < /proc/version && echo $A
(string manipulation can get the other options)

ps needs looking at /proc/[0-9]*/status and each corresponding /proc/[0-9]*/cmdline

cpuinfo == cat /proc/cpuinfo

free ~= cat /proc/meminfo

lsmod == cat /proc/modules

lspci ~~ cat /proc/bus/pci

lsusb ~~ cat /proc/bus/usb

mount with no parameters (list mounts) is ~~ cat /proc/mounts

and rather than using cat which is external to some shells and slower
read LINE < /proc/file ($LINE will be the first line)

while read LINE || [ "$LINE" ]; do echo $LINE; done < /some/file
(or you can replace echo with parsing, commands, etc)

.
.
.
.
I'm not saying C and executables are slower than a script, just that it is sometimes faster to use a builtin than to even call up an external command ... If you were inside of a C program the same would hold true ... hmm I wonder, has anyone written an init script in C? I thought about it (which is where waitpids camefrom) but ... I find myself reinventing the wheel a lot (granted, a wheel with less rotating mass usually).

[goingnuts]

This is really awesome, I'm glad it's still around (first post from 2010?)

Thanks! Me too...

I find myself reinventing the wheel a lot (granted, a wheel with less rotating mass usually).

If no one did, what you do, wheels would still be made of stone.
Further tests done: Now have disktype elspci fusermount guess_fstype setserial hotplug2stdout cddetect hotplug2stdout_notimeout mount-FULL umount-FULL losetup(-FULL) puppyinputdetect unsquashfs3.4 mksquashfs3.4 in place saving more than 50% of total size - and more to come.
Had to shift to uclibc (from dietlibc) but the potential of applications to compile static is larger (although resulting bins are also larger..)
Your hints on using /proc and /sys info are good - elimination is by far more effective than reducing size of unneeded bins...
Any thoughts on kernel-module size-reduction? Or kernel-size reduction?

[PANZERKOPF]

disktype (from guessfstype?) there was a patch for blkid that does the same thing, but

Busybox blkid shows only LABEL and UUID fields, whereas "full" blkid shows also TYPE field. Will be shown that field after applying a patch?

Or kernel-size reduction?

Sometimes I think about 2.4 version of kernel...
Version 2.4 is smaller (and maybe faster) but hasn't many drivers for newest hardware.

[technosaurus]

re: kernel size / module size

you can add a "-s" and/or rip out the "-g" to CFLAGS when building the kernel (this will strip all of the .o objects as they are made ... but I think this may happen in the kernel already with an option)

Denys Vlasenko @ busybox has patches to allow --gc-sections linking, but my experience has shown that this is incompatible with modules/plugins (but DV is clever and maybe it works, ... I haven't looked closely)

some kernel modules can be stripped with:
strip [--strip-|unneeded|debug] -R .note -R .comment *.ko
(it doesn't _always_ work, so you may need a blacklist for the unneeded/debug parts )

busybox modutils allows xz compression ... but really using compressed files in a compressed file system is usually a bad trade ... better to use squashxz and leave them alone, this will allow the squash compressor to find common strings among them ... like kprintf etc...

_never_ upx busybox (or other binaries where speed is important) , the speed penalties aren't worth it and again the size savings on a compressed file system are negligible at best and often negative

to compress an initrd with xz (kernel must support ... needs 2.6.28+, or a patch)
cd /path_to/init_tree
find . | cpio -o -H newc | xz -ze9 >../initrd.xz
(it may be easier to have a .gz extension though unless you feel like editing a bunch of stuff)
Edit- you can use an xz compressed init with a .gz extension the kernel doesn't care about the extension name, but scripts do and may specifically look for initrd.gz

but there is actually a better way - the kernel will let you specify a path to the initramfs so that it gets built in to the kernel image
(you can specify a precompressed image to but then it has to be uncompressed twice for no real benefit)
if you need to update it you can either keep the kernel around and rebuild or use an initrd.gz that will overlay (overwrite) the existing internal files
(I used to debug/improv/test the init scripts, by loading a second initrd image called myinit.gz which only contained the modified script)
If you distribute with the initramfs built in, it would be a good idea to still distribute an unused initrd.gz that just has a README

as far as kernel options go, you can now use the ext4 code for ext2 and ext3
as you have already done, eliminate modules that are <almost> always used by including them in the kernel image
if you can get away with only reading ntfs, use the kernel's and load ntfs-3g only when write access is needed
Edit

Code: Select all

#kernel params worth testing out
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_KERNEL_XZ=y
CONFIG_DEVTMPFS=y

CONFIG_DEVTMPFS_MOUNT=y
#or use busybox mount? need to test

EXTRA_FIRMWARE="space.bin separate.bin list.bin of.bin firmware.bin"
#useful for the firmware you would have in the initramfs

CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE="/mnt/sda1/initramfs"
#I keep my initramfs there, change to where you have yours

http://cateee.net/lkddb/web-lkddb/DEVTMPFS_MOUNT.html
and http://cateee.net/lkddb/web-lkddb/DEVTMPFS.html
http://cateee.net/lkddb/web-lkddb/EXTRA_FIRMWARE.html
see also http://cateee.net/lkddb/web-lkddb/FIRMW ... ERNEL.html

it would be really cool if the bulk of the init process could happen in an rxvt terminal from within jwm (+Xvesa)
at a minimum you would need to enable devtmpfs (this would also allow us to drop some of the static /dev nodes) because rxvt needs to be able to access /dev/pts/%i (i fought with trying to do this for a while, but not with devtmpfs)

Busybox blkid shows only LABEL and UUID fields, whereas "full" blkid shows also TYPE field. Will be shown that field after applying a patch?

yes, but Denys wanted to the submitter to use pointers, rather than duplicating code (it added unnecessary size) I understand, the whole pointer thing still gets me confused from time to time (well usually actually)

[goingnuts]

technosaurus:Excellent sum up of what to do and what not to do. Thanks!
I have used some days trying to merge squashfs-tools from v.3, v3.4 and v4.0 renaming almost every function, variable and constant in all of them - only to realize that the source code need to be split to make it work (multiple defined problem). But after giving up also realized that to cover pupngo with kernels from P216, P412, P430 and P525, you only need mksquashfs3.0, mksquashfs4.0 and unsquashfs4.0 (and I can merge mksquashfs3.0 and mksquashfs4.0). Reduced size from 917K to 455K (120K upxed).

Then almost made it with cdrkit-1.1.10 but got stuck in the below function from librols/snprintf.c where the missing function is format:

Code: Select all

#ifdef	PROTOTYPES
EXPORT int snprintf(char *buf, size_t maxcnt, const char *form, ...)
#else
EXPORT int snprintf(buf, maxcnt, form, va_alist)
	char	*buf;
	unsigned maxcnt;
	char	*form;
	va_dcl
#endif
{
	va_list	args;
	int	cnt;
	_BUF	bb;
	bb.ptr = buf;
	bb.count = maxcnt;
#ifdef	PROTOTYPES
	va_start(args, form);
#else
	va_start(args);
#endif
	cnt = format(_cput, (long)&bb, form,  args);
	va_end(args);
	if (maxcnt > 0)
		*(bb.ptr) = '\0';
	if (bb.count < 0)
		return (-1);
	return (cnt);
}

Anyone have an idea about which header to add or which library to link with?

[technosaurus]

snprintf is probably part of libc already, if so, you can probably comment it out.... or in this case maybe just add -DPROTOTYPES to CFLAGS

depending on the error (incompatible or multiple definition?)

[goingnuts]

The error is:

Code: Select all

 In function `snprintf':
snprintf.c:(.text.snprintf+0x28): undefined reference to `format'

Tried the -DPROTOTYPES but no change...
I can comment the line out, supply a snprinf.c from musl or totally exclude the snprintf from the objects being used - and finish the build...but don't know if its ok - the cdrkit-1.1.10 source tree is a little scattered and the comment in the header of the snprintf says:"This file has been modified for the cdrkit" suite.

Guess I just have to test the builded wodim with some burns.

Anyway - the multicallbins (mcb´s ?) are really great - have seen size reductions of 60% which almost equals to size of dynamic linked bins (without counting the size of the needed libs in).

[technosaurus]

Not all libc's have snprintf, so normally autotools builds this into either config.h or the makefile, so that it only builds on systems that don't have it. But as I am sure you have discovered, autotools suck... which is also why I used a standard shell build script (i would use makefile(s) if it grows much larger so that libs/objects get rebuilt iff changes occurred)

I haven't been working much on compiling and mcbs lately, b/c I am trying to write up some patents that involve using a device manager, so I have been studying mdev, udev, hotplug etc... and writing my own implementation in shell. This was helpful:
http://www.bangmoney.org/presentations/hotplug/
I expect that much of this will make its way into jwm_tools and bashbox for drive insertion etc... it turns out that it may even speed up the whole process because the previously mentioned tools run commands via sh -c <command> <args> but if I can do all of it in a shell using builtins to begin with it won't need to call another shell AND it won't use bash when it is linked to sh. I just wish I had more plugable devices to test.

[goingnuts]

...I am trying to write up some patents that involve using a device manager, so I have been studying mdev, udev, hotplug etc... and writing my own implementation in shell.[...]
I just wish I had more plugable devices to test.

Sounds good! If one could profile a plugable device (catch all generated files in proc/sys) and afterwards simulate insertion on another machine you could have access to plenty of plugable devices - but dont know if its possible.

[PANZERKOPF]

If one could profile a plugable device (catch all generated files in proc/sys) and afterwards simulate insertion on another machine you could have access to plenty of plugable devices - but dont know if its possible.

Catching can be easily performed but seems we cannot manually create or delete something in /sys directory to simulate nonexistent device.

[goingnuts]

Maybe a simulated input could just be read from a different directory (a fake /sys or /proc)?

Further on the mcb´s: Created attached mcb of Ace-of-Penguins1.3 - containing the games canfield, freecell, golf, mastermind, merlin, minesweeper, pegged, solitaire, spider, taipei and thornq (and additional the taipeditor taipedit). All the games uses libcard with is statically included. Upxed bin is 104K - 184K unpacked. Normal build would be 250-297K.
The mcb can be anywhere in your path and installed by command: "mcb_ace --install" which will create the needed symlinks (not needed for the attached pet though...).
Update 20190908: The original posted mcb of ace-1.3 is dynamic linked. The patch to give the mcb was lost and I recently recreated it and also updated relevant things from version 1.4. So attached my patch as well as a static linked version packed as pet. Should work on any Puppy version.

[goingnuts]

Another mcb - merging almost all bins from initrd.gz into one bin + some needed in main sfs for start up (udevd a.o.).
Contains:
udevd fuser puppyinputdetect guess_fstype hotplug2stdout hotplug2stdout_notimeout cddetect setserial fusermount elspci modprobe.bin modinfo ntfs-3g ntfs-3g.probe disktype resize2fs mke2fs mkfs.ext2 mkfs.ext3 fsck e2fsck fsck.ext2 fsck.ext3 find cp mount-FULL umount-FULL losetup rev
in 740K static build - thats more or less same size as dynamic linked bins exclusive needed libs. All programs are the full versions.

Place mcb_init in /bin and use command:"mcb_init --install" to create the symlinks. Symlinks are hardcoded to be created the usual places (/bin, /sbin, /usr/bin or /usr/sbin).

Its experimental and contains some dangerous disk tools - so use in non-critical environment. I did test most of the stuff but as always there is no guaranties.

[technosaurus]

Wow, you got ntfs3g too, I tried busybox-ing that one a year or so ago, and it almost seemed to be incompatible with itself. Excellent work.

[PANZERKOPF]

MCB is interesting idea. Thanks!

Another mcb

Please can you share a source of some MCB you have made? I want to learn its structure.

[technosaurus]

the last updated source that I made (Xapps) was here:
http://www.murga-linux.com/puppy/viewto ... h&id=46813
as it grows the build infrastructure will need to grow with it though, that one was just jwm, rxvt and getgui

Edit: I have been pondering the possibilities that open up with these multicall binaries. If we carefully select only apps with portable code, and used llvm, we could have a universal linux that converts the llvm bytecode to optimized machine code at first boot. (This is much more difficult if you try to use shared libraries). With the fluid and varied nature of the ARM (and other) markets, I couldn't think of a more effective way to keep up.

[goingnuts]

technosaurus: Thats beyond my capabilities to do or understand (for now..). I have been working on "pUPnGo-from-source" for some time and are more or less done - could that be a starting point for your idea?

Further findings with the mcbs: Great potential to decrease size on disk especially compared to single build static bins. I think one have to think carefully of which bins to bundle - maybe not small often used bins with large seldom used bins.
For dyn linked mcb it seems more effective to bundle bins with equal deps as the mcb will have the sum of all included bin's deps. Here it might give benefit to partly include some libs static to reduce the total shared libs needed.
There might be some memory usage considerations but I haven't proved that yet.

It seems that every lib you link the mcb with at compile time will become a dep even though its not needed at all - for shared libs and static lib builds - so carefully evaluate every single application for really needed libs before introducing it into the mcb.

For mcb's using shared libs the size reductions is not so obvious as reduction mainly seems to come from equal functions between the different apps build in. If one could equalize some common used functions between apps (usage(), getopts() a.o) the size reduction would probably be more pronounced - but what a h... of work to do.

One other general benefit of the mcb's are the much more simple file-layout (reductions of bins present). I have reduced my initrd.gz to hold only two real bins (busybox and mcb_init) and chosen to make all BB/mcb symlinks via first thing done in init - but that might be too extreme as it just satisfy the demand for smallest size on disk.

The mcb-bins reduces well using upx and I haven't seen tremendous performance slowdown doing this but it might show up on not so fast processors.

So all in all: mcb are a very attractive tool if balanced for your need and target properties - that is - use with care

One final thought: If a more general interests in this mcb-thingy is present we might need a more user/developer-friendly approach - like build-roots/kernels "make menuconfig"...but that might be an other full-time new project...

[technosaurus]

I was once again playing with compiler flags (this time -fvisibility=hidden) and I keep getting back to this multicall binary concept - You get ~10% size reduction if you keep the object files in a folder and link directly to the objects. In other words if you compile the object files to ./libjwm.a/*.o (NOTE that libjwm.a is a directory name) rather than using ar to combine them into an object archive, then the symbol table is essentially eliminated (though some libraries will try to fight you with __attributes...). This is not a problem since we won't need symbol tables. I am all for the idea of starting a separate project for it, send me a PM if you get a chance.

[goingnuts]

Attached static mcb holding Xvesa/xinit/rxvt. Unpacked size 690K/upxed 329K (shipped as this).
Essentially it can replace your Xvesa+xinit+rxvt. Place the binary in /usr/X11R7/bin, make a backup of your original Xvesa/xinit/rxvt (rxvt might reside in /usr/bin so rename that one), run mcb_vesa --install at a command prompt and restart Xserver to use the new installed versions.
Feedback from test welcome.

I could have included jwm as well but jwm combines better with other image-lib dependent applications or use the static version shipped a few post earlier.