Alternative way to build Ubuntu / Debian Puppy [RETIRED]

[mavrothal]

Looks good. I really wonder how big it is with all Puppy applications, before cleanup.

With firefox is ~170MB (gzipped SFS) Abiword/Gnumeric and friends pulls a lot more though.

[jamesbond]

Looks good. I really wonder how big it is with all Puppy applications, before cleanup.

With firefox is ~170MB (gzipped SFS) Abiword/Gnumeric and friends pulls a lot more though.

That's because Abiword and Gnumeric depends on GTK3 So for these kind of apps, you need to build GTK2 versions and package them into DEB and upload to own repo. And you can bump up the compression by using "-comp xz -Xbcj x86" instead of the default "-comp gzip" I have in my sample pkglist. It will still be big, though.

Another trick is this - you can attempt to turn off dependency (%nodepend) and install the package and its dependencies manually, with you choosing only the deps you want to install (this is no fun, but works). Or, you can keep dependency checking on, but later you %remove the packages that you don't need. Some of the dependencies are really not needed (e.g. udev pulls a crap load of dependency, which I ignore with %nodepend, and it still works).

Of course, the success of this depends on the package itself. With some packages, the depedencies are really with loadable modules/plugins - if you don't include the dependencies, those plugins won't work but the main application still work. Some packages, however, have dependencies compiled built into them - with these kind of packages you're really out of luck.

With abiword/gnumeric which depends on gtk3, you don't really have an option other than pulling all those gtk3 stuff I'd recommend making our own DEBs for these as they are not core to the system. (Xorg, however, is core, and we should keep parent distro's one).

# grep systemd repo-trusty-i386/pkgdb | wc -l
58
# grep libgtk-3 repo-trusty-i386/pkgdb | wc -l
540
# grep libgtk2 repo-trusty-i386/pkgdb | wc -l
1329

I'm very troubled with the fact that synaptics depends on gtk3, I hope there is a way to build it to depend on gtk2. Aptitude-gtk is dead, and aptitude-ncurses depens on .... python?! Any other GUI front-end to apt?

Also, rootfs-skeleton is a mess. It clutters the main SFS with so many files.

Don't like it? Here's a trick: Mavrothal asked me to add support for installing rootfs-packages, which I did. I now have another purpose for that "rootfs-packages" - how about we make a package called "myown-rootfs", and in the pkglist we don't do %addbase but instead to "%addpkg myown-rootfs"
I had an example called james-mods which replaced xwin with a simpler variant, and replaces xorg-wizard, but nothing is stopping you to supply a full rootfs there I you look inside deb-build.sh, the code handling %addbase and %addpkg is identical

[mavrothal]

With abiword/gnumeric which depends on gtk3, you don't really have an option other than pulling all those gtk3 stuff I'd recommend making our own DEBs for these as they are not core to the system. (Xorg, however, is core, and we should keep parent distro's one).

# grep systemd repo-trusty-i386/pkgdb | wc -l
58
# grep libgtk-3 repo-trusty-i386/pkgdb | wc -l
540
# grep libgtk2 repo-trusty-i386/pkgdb | wc -l
1329

I'm very troubled with the fact that synaptics depends on gtk3, I hope there is a way to build it to depend on gtk2.

According to the synaptic home page it needs gtk+2.4 (or latter).
But distro compatibility and size will be a continuous battle that we can not win with 2-3 people behind puppy. Specially if we do not want to employ BK's elaborate "templates" system and mixing distro and non-distro binaries.
But I think for now lets try to get a fully blown system and see how does it do size, efficiency and functionality wise.
BTW would be handy to either utilize the puppy kernels (since the build scripts are there too) or provide a 32bit "fatdog-like" kernel (sorry too little time these days )

[01micko]

Perhaps puppy typical gtk2+ programs (ie compiled light as possible) could go in a separate sfs (adrv?) and therefore not registering with dpkg. However that recreates an age old problem of stuffs in sfs not registering with package management. That could be addressed in a native package manager but then that defeats the purpose of this project.

The end user doesn't give a toss. Well, they toss the CD if it doesn't "just work". Likewise they don't want to know about terminals. A gtkdialog or gtkbuilder frontend for apt? Sounds scary/crazy eh?

[jamesbond]

Specially if we do not want to employ BK's elaborate "templates" system

I don't know what you mean by this. What is this "template" system?

and mixing distro and non-distro binaries.

I think this can't be avoided. Xdialog and gtkdialog already comes from ourselves. Synaptic will probably have to be our own too (thanks for pointing this out), and so will abiword/gnumeric be if we want to have them in reasonable size. But they must be done in the proper native package format so they get recorded. DEBs for Debian/Ubuntu, TGZ/TXZ for slackware, etc.

BTW would be handy to either utilize the puppy kernels (since the build scripts are there too) or provide a 32bit "fatdog-like" kernel

We don't need fatdog-like kernel, I did it just so I can quickly come up with a script that builds an ISO. Puppy's traditional initrd has a mixture of kernel modules in initrd and in basesfs. This is complicated, we need to copy some modules to initrd, do depmod, then copy the rest and firmware to basesfs, etc. This can still be done (although I don't provide the tool to do that), and the resulting puppy will still work.

So to simplify matter, I put *ZERO* modules in initrd. That's right, there is no modules in initrd. This makes the initrd, in a sense, "universal"; it can be used for any puppies. I use ZDRV exclusively to load kernel modules (kernel-modules.sfs). Basesfs (puppy.sfs) is free from kernel modules, and thus is universal too. If you want to update kernel, just update vmlinuz and kernel-modules - then you're done. No complicated unsquashfs/mksquashfs, no uncomplicated extracting initrd and rebuild.
For this to work, the kernel must be compiled with enough kernel modules to be able to find ZDRV and load it. It means loopback, aufs, squashfs, filesystem drivers, usb drivers (if we want to boot from usb) *must* be compiled to the kernel, not as a module. That's all there is to it. If puppy's kernel is currently compiled with that, then it should be straightforward to download puppy's kernel and build the ISO. Otherwise, either the kernel config needs to be modified, or we need to use standard puppy initrd build as above.

Perhaps puppy typical gtk2+ programs (ie compiled light as possible) could go in a separate sfs (adrv?) and therefore not registering with dpkg. However that recreates an age old problem of stuffs in sfs not registering with package management.

I personally think there isn't a problem with that. Apps in SFS, by its nature, ephemeral, they can be added or removed at anytime. So I don't see the need for them register with the native package management, and in general, it is not good for regular programs to depend on presence of such SFS.

That being said, if you really want to integrate them, for dpkg this is going to be a bit difficult (not impossible, but difficult). In slackware system, the installed packages are recorded in individual manifests in /var/log/packages. To give an appearance of an "installed" package, all you need to do is add one more package manifest in that directory and that package will be "installed." In dpkg, this is not that easy - packages are recorded in two places, one individual manifest like slackware in /var/lib/dpkg/list, one in big giant database called /var/lib/dpkg/status. It is the latter which gives us headache. But it is still possible - however, one needs to agree on having a "post-load" and "pre-remove" scripts to be run on SFS load/removal; and that script will be responsible to manipulate /var/lib/dpkg/status.

That could be addressed in a native package manager but then that defeats the purpose of this project.

A package manager which doesn't come from the parent has two problems.
1) It always suffers from "translation" problem. And as usual when translating something, some information or some context may be lost, which *may* cause issues later. That's the reason to stay with parent distro's manager.
2) Then there is this "current-ness" problem. How to know whether the cached, translated package database is not out of date compared to the parent distro's native package database?
Both of these are not intractable problems, and can be addressed (as the current PPM obviously shows), but they are still quite challenging nonetheless.

The end user doesn't give a toss. Well, they toss the CD if it doesn't "just work". Likewise they don't want to know about terminals. A gtkdialog or gtkbuilder frontend for apt? Sounds scary/crazy eh?

Not scary, but with the number of packages around ~44,000, you do need crazy optimisation to make gtkdialog fast

[Iguleder]

Makes sense.

So ... now it's time for us to set up an Apt repository?

There's an alternative - merging Puppy-specific packages into rootfs-skeleton.

[stemsee]

For this to work, the kernel must be compiled with enough kernel modules to be able to find ZDRV and load it. It means loopback, aufs, squashfs, filesystem drivers, usb drivers (if we want to boot from usb) *must* be compiled to the kernel, not as a module. :

What is the reason for not having all modules built-in to the kernel? The kernel should have better performance too.

[Keef]

Doesn't "localyesconfig" select the modules just for the current running system?

[Iguleder]

For this to work, the kernel must be compiled with enough kernel modules to be able to find ZDRV and load it. It means loopback, aufs, squashfs, filesystem drivers, usb drivers (if we want to boot from usb) *must* be compiled to the kernel, not as a module. :

What is the reason for not having all modules built-in to the kernel? The kernel should have better performance too.

They're huge - a kernel package is ~50 MB with the modules. This means, the distro RAM usage starts at the uncompressed size of the kernel package.

Having all the modules in the main SFS doesn't have any downsides. A kernel with all the modules jamesbond specified should be 2-3 MB.

[stemsee]

yep, make sure everything is on! ... then after run make menuconfig and modify accordingly! Might save some time!

[gcmartin]

Questions:

• In 2014 is GTK3 all that bad for this new build tool?
• Would life be simpler for the march into the future if we embraced it now?

Everyone knows I am not a developer, merely a testor-user.

Curious

[mavrothal]

Just generated the woof-next branch in woof-CE.
Basically moved all woof building scripts to woof-code/woof2-scripts_and_files and changed merge2out not to copy over all the distro-package-...
It does not have jamesbond's changes so it can not build anything right now, but does have jamesbond as a woof-CE member

[01micko]

Soon I'll work on modifying kernel kit to package the modules in an sfs (zdrv) so we can make use of the simple fatdog style initrd. Anyone think of anything else to be into the kernel image? So far we have squashfs, aufs, ext2,3,4, .....other fs? Really "other fs" can be up to the builder; eg: if one wants to try booting from hfs+ then that must be a builtin. Same for reiser, btrfs, f2s etc.

[mavrothal]

Soon I'll work on modifying kernel kit to package the modules in an sfs (zdrv) so we can make use of the simple fatdog style initrd. Anyone think of anything else to be into the kernel image? So far we have squashfs, aufs, ext2,3,4, .....other fs? Really "other fs" can be up to the builder; eg: if one wants to try booting from hfs+ then that must be a builtin. Same for reiser, btrfs, f2s etc.

VFAT for sure.
f2fs if possible too.
What about booting from NTFS volumes?
Although is sacrilegious to run linux in a Mac ( ) there is a bunch of 32bit intel core duo (not core2duo) Macs that are stuck with OSX 10.6 and not supported anymore. So go for it.
I guess the other question is SCSI/IDE/SATA etc.

Another option would be to have the entire kernel in the initrd so you do not really need to wary about what is where. XOpup does that but it has really few modules.
Should not really affect boot times since this file will be loaded from somewhere and will likely decrease the kernel memory since a bunch of unused modules will not be in vmlinuz.

[Iguleder]

All we need is:
- Drivers for USB mass storage devices
- All hard drive drivers (they're tiny)
- SCSI CDROM drivers
- Drivers for all file systems we want to be able to boot from - ext2, ext3, ext4, FAT16/32, NTFS (?), XFS, reiserfs (for unlucky users who installed an old distro and kept the file system when they moved on to Puppy) and btrfs ... isn't f2fs too young?
- Squashfs and Aufs

This is the classic Puppy recipe - it should produce a 2-3 MB kernel.

I think sticking to a LTS kernel is the way to go, because pretty often, Aufs gets messed up with the latest and greatest (segfaults, in the kernel ). I've been using 3.10.x since it received its LTS status and it's simply awesome.

EDIT: 01micko - here's my kernel building script for 3.10. It's very Puppy-like, except the fact it's for Linux-libre and doesn't have Aufs. It took some time and additional gray hair to get UEFI to work (i.e, it boots but you get a black screen because it doesn't have the UEFI framebuffer module ), so it's a good reference configuration.

EDIT 2: I have a static toybox and many static daemons I wrote on my own (i.e syslogd and klogd). Functionality-wise, they should be compatible with the BusyBox ones. Maybe we could use them in the initrd instead of BusyBox. They're much lighter and smaller.

[01micko]

Thanks Iguleder - using your DOTconfig with 3.12.21 with aufs added. Just raw, as in I haven't checked if new modules are available. Compiling now.

I'll add James' firmware to a kernel-modules.sfs and see if the thing boots - more soon.

EDIT: BTW 3.12 is LTS too and what slacko64 was using before the 3.13 experimental release.

EDIT: failed to boot, initramfs problem, anyway rebuilding using fatdog-630-rc2 config ..which is what i had on hand ..should "just work"

EDIT: FD config booted but no keyboard at console for me. I honestly don't know if it worked in fatdog itself at console on this machine so I'll have to check. (LATER: checked .. yes it did work in FD-630-rc2 k3.12.6)

NEXT Morning
EDIT: depmod call in /etc/rc.d/rc.sysinit fails so that's why kernel modules didn't load. I ran depmod on the kernel modules before squashing and now it works . I should be able to mod kernel kit appropriately to produce a fatdog style kernel.

Code: Select all

Linux puppypc29976 3.12.21 #1 SMP Sun Jun 8 07:25:04 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/DISTRO_SPECS
DISTRO_NAME='ubuntu Puppy'
DISTRO_VERSION=7.0
DISTRO_BINARY_COMPAT='ubuntu'
DISTRO_FILE_PREFIX='ubuntu'
DISTRO_COMPAT_VERSION='ubuntu'
DISTRO_XORG_AUTO='yes'
DISTRO_TARGETARCH='x86'
DISTRO_DB_SUBNAME='ubuntu'
DISTRO_PUPPYSFS=puppy.sfs
DISTRO_ZDRVSFS=kernel-modules.sfs

Ignore the $DISTRO_TARGETARCH.. uname certainly wants us to know that this is x86_64, 3 times

[01micko]

I just did a fresh install of slacko-5.7 to test out kernel-kit modifications on 32 bit and the Ubuntu install using tahr bins. I have just pulled james' latest so it will have the mods he put in (so far I haven't updated since initial pull).

If all goes well I'll post kernel kit here for testing and if that goes well I'll push it to woof-CE/testing. Kernel is compiling now. (again 3.12.21). Kernel kit will have both configs. BTW, it's 4G not PAE so I can test on my old centrino lap top. That can be easy changed in the config for those who like PAE, and beauty of the FatDog kernel method is simplicity in switching kernels. Just replace vmlinuz and kernel-modules.sfs and reboot.

Mavrothal.. it might actually be time soon for a new git repo.. woof-NG

[stemsee]

This new kernel, initrd and kernel-modules.sfs combination, has to be the most significant breakthrough in Puppy Development for ages. The time I have wasted fiddling around with DISTRO-SPECS in main.sfs and initrd and swapping modules around and separating modules for initrd, and/or compressing/decompressing *.ko.gz mods, and all the other stuff was utterly unnecessary! And this method will, hopefully, prove it!

Kudos to you. Anything I can do let me know!

Edit: Any chance of ensuring all puppys build with the ability to 'xinput create-master second' , only Lighthouse64 has that ability and after research and experimentation I can shed no light no the issue. It is the one thing which gives superiority to Debian based systems in my opinion.

[01micko]

Ok, 32 bit kernel is a success and posting from it now, just added dhcpcd to the default, hacked build-iso.sh to grab the kernel from my personal repo and added firefox and we are away.

Code: Select all

# uname -a
Linux puppypc27288 3.12.21 #1 SMP Sun Jun 8 15:24:18 EST 2014 i686 athlon i686 GNU/Linux

More later.. dinner time!

[01micko]

Time for some code

Diff to build iso (grabs kernel)

Code: Select all

--- build-iso.sh.orig	2014-06-03 20:54:06.660580810 +1000
+++ build-iso.sh	2014-06-08 18:02:47.560002531 +1000
@@ -10,6 +10,7 @@ ISO_ROOT=${ISO_ROOT:-$OUTPUT_DIR/iso-roo
 PUPPY_SFS=${PUPPY_SFS:-puppy.sfs}
 KERNEL_VERSION=${KERNEL_VERSION:-3.12.9}
 PARENT_DISTRO=${PARENT_DISTRO:-ubuntu} # or debian
+KERNEL_URL=${KERNEL_URL:-http://distro.ibiblio.org/fatdog/kernels/700}
 
 WOOF_ISO_ROOT=${WOOF_ISO_ROOT:-boot}
 WOOF_INITRD=${WOOF_INITRD:-boot/initrd-tree0}
@@ -29,7 +30,7 @@ install_boot_files() {
 install_kernel() {
 	for p in vmlinuz kernel-modules.sfs; do
 		! [ -e $ISO_ROOT/$p ] &&
-		wget -P $ISO_ROOT -c http://distro.ibiblio.org/fatdog/kernels/700/$p-$KERNEL_VERSION
+		wget -P $ISO_ROOT -c $KERNEL_URL/$p-$KERNEL_VERSION
 		mv $ISO_ROOT/$p-$KERNEL_VERSION $ISO_ROOT/$p
 	done
 }
@@ -58,7 +59,7 @@ make_iso() {
 	-volid "Puppy-Linux" \
 	-iso-level 4 -D -R  \
 	-b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table $ISO_ROOT/
-	isohybrid -o 64 "$OUTPUT_DIR/$OUTPUT_ISO"
+	isohybrid "$OUTPUT_DIR/$OUTPUT_ISO" #-o 64
 }
 
 ### main
# 

NB: The third hunk of the diff will not be necessary for FatDog (isohybrid part)

Apply that and run with KERNEL_URL as an arg (will work as I have uploaded that kernel)

Code: Select all

KERNEL_VERSION=3.12.21-slacko32FD4G KERNEL_URL=http://01micko.com/packages/ ./build-iso.sh

I'll attach kernel-kit adjusted to build a fatdog64 (or 32) bit kernel.