Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Compatible Puppy utilities

#161 Post by gyro »

I've uploaded several pets for various Puppies.
These contain patched versions of /etc/rc.d/rc.shutdown, /etc/rc.d/rc.update, /usr/local/petget/installmodes.sh and /usr/local/pup_event/frontend_funcs.
They are patched to be compatible with both "overlay" and "normal" Puppies.
I have published these pets so that these can be tested in "normal" Puppies.
When installed in the corresponding "normal" Puppy they should make absolutely no difference.

Download:
for artfulpup-17.11+16.iso http://www.fishprogs.software/puppy/ini ... ul-0.1.pet.
for LxPupSc-18.04T-k64 http://www.fishprogs.software/puppy/ini ... sc-0.1.pet.
for slacko-700-RC3_k4.4.70-PAE_2018-04-05 http://www.fishprogs.software/puppy/ini ... ko-0.1.pet.
for slacko64-6.9.9.9-uefi_r6048 http://www.fishprogs.software/puppy/ini ... 64-0.1.pet.
for Upup_Bionic_Beaver-18.04+5 http://www.fishprogs.software/puppy/ini ... bb-0.1.pet.
for xenialpup-7.5-uefi http://www.fishprogs.software/puppy/ini ... al-0.1.pet.

I've also uploaded my proposed patches against the current 'testing' branch as a set of diff files.
Download http://www.fishprogs.software/puppy/ini ... ffs.tar.gz.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#162 Post by gyro »

I have attached "install2usb3.diff".
Changes from previous patch:
1. It only writes "pboot" boot parameters if there are 2 partitions.
2. Bump the version to "0.4.3"

gyro
Attachments
install2usb3.diff.gz
gunzip to produce install2usb3.diff
(2.38 KiB) Downloaded 212 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

FYI

#163 Post by gyro »

I've recently installed an "overlay" Puppy to a slow usb stick using "install2usb", and choosing "vfat" so I end up with just a single vfat partition on the usb stick.
I found the installed Puppy to be quite responsive when run with the "Archive" save mechanism.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#164 Post by gyro »

I have attached "overlay_installers-0.7.sfs".
This contains my latest hack of 01micko's "install2usb".
If the target "initrd.gz" does not contain "OVERLAY_SPECS" then it appends a "TZ=" boot parameter representing the current timezone of the host Puppy.
I've realised that the current "init" in the woof-ce 'testing', will set the timezone if "TZ" is set.
So the "TZ=" boot parameter will correctly set the time during the execution of the target "init" and so avoid any possibility of fsck reporting mount time errors.

I have also attached "install2usb4.diff", which is the current patch to 01micko's code to produce my hack.

gyro
Attachments
overlay_installers-0.7.sfs.gz
gunzip to produce overlay_installers-0.7.sfs
(9.72 KiB) Downloaded 218 times
install2usb4.diff.gz
gunzip to produce install2usb4.diff
(2.43 KiB) Downloaded 221 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#165 Post by gyro »

I have attached a further hacked version of the "change_kernels" script.
The previous version sometimes tried to "umount" a busy partition that it had not mounted, this version does not.

I have also attached "change_kernels2.diff", this patch is supposed to change 01micko's released version to my current hacked version.

gyro
Attachments
change_kernels.gz
gunzip to produce "change_kernels"
(4.4 KiB) Downloaded 228 times
change_kernels2.diff.gz
gunzip to produce "change_kernels2.diff"
(1.87 KiB) Downloaded 223 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#166 Post by gyro »

A simple fix to "install2usb", change:

Code: Select all

printf "1\ny\n"| fsck.vfat $FIXFATPART
to

Code: Select all

printf "1\ny\n"| fsck.vfat -r $FIXFATPART
"-r" is the default in recent versions of "fsck.vfat" but not in older versions.
I tested "install2usb" on Tahr 6.0.6 and the vfat partition was not fixed, until I made this change.
The changed code still works in recent versions.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Indicating that vmlinuz is in differnt partition

#167 Post by gyro »

I'm having second thoughts about how installers like "install2usb" should indicate that "vmlinuz" is on a partition other than the one containing the frugal install directory with the sfs files.
The problem is that while the boot parameter method works fine in the running target Puppy, it does not help a script that's running on a host Puppy, but trying to update the frugal install directory of a target Puppy, e.g. "overlay-setup-frugal".

My alternative would be for the installer to write a file called "pbootlocation.txt" (or "puppyboot.txt") in the frugal install directory alongside the sfs files.
This file would contain something like the following:

Code: Select all

BOOTPART='VFAT'
BOOTDIR='/xenialpup-7.5'
Where 'VFAT' is the label assigned to the vfat partition, and '/xenialpup-7.5' is the sub-directory of that partition containing "vmlinuz".

This approach should work for any script processing the frugal install directory, simplifies the code in the installer, and does not involve changing "init" or "PUPSTATE".

The aim of either approach is to support not only the "install2usb" script, but also setups where the the boot partition is a usb device but the Puppy frugal install directories reside on a different device, e.g. a Linux partition on an internal HD.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

"vmlinuz" not in frugal install directory issue

#168 Post by gyro »

"vmlinuz" not in frugal install directory issue:
There is a simpler way.

"install2usb" doesn't do anything about the the issue.
"change_kernels" looks for "vmlinuz" in the frugal install directory, if it doesn't find it, it opens a "select file" dialog, so the user can locate it.
Silimarly, if it doesn't find a "zdrv...sfs" in the frugal install, it opens a "select file" dialog so the user can locate it.

This approach can even cope with the situation where the user has "specified" the "zdrv...sfs" to be a file with a different name, e.g. "/mnt/sdc7/kernels/kernel_modules-4.9.92.sfs".

But if the frugal install directory is "normal", then things proceed as before.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#169 Post by gyro »

I have attached:

"overlay_installers-0.9.sfs", which includes my latest hack of 01micko's "install2usb", and "overlay-setup-frugal".
This "install2usb" does nothing about the "vmlinuz" in a different partition issue.

"change_kernels" the corresponding hack of this script.
If it can't find "vmlinuz" in the frugal install directory and it's a usb device, it also looks in PSUBDIR on the 1 partition of the device.
Otherwise it opens a file selction dialog for the user to specify the location of the appropriate "vmlinuz".

"install2usb9.diff" my patch to "install2usb".

"change_kernels9.diff" my patch to "change_kernels"

gyro
Attachments
overlay_installers-0.9.sfs.gz
gunzip to produce "overlay_installers-0.9.sfs"
(9.86 KiB) Downloaded 480 times
change_kernels.gz
gunzip to produce "change_kernels" script
(4.53 KiB) Downloaded 460 times
install2usb9.diff.gz
gunzip to produce "install2usb9.diff"
(2 KiB) Downloaded 481 times
change_kernels9.diff.gz
gunzip to produce "change_kernels9.diff"
(2.04 KiB) Downloaded 476 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Release of overlay_init-0.9

#170 Post by gyro »

Release of overlay_init-0.9

The version jump to 0.9 is to indicate that this project is almost done.
I don't expect the "init" script to get any more significant changes.
The utilities are now pretty much functionally complete.

Unfortunately this project has become a little muddied by some of the existing Puppy files that it replaces, changing in woof-ce 'testing'.
To ensure that the patched version in this project properly reflects the version in the parent Puppy, there are multiple releases, each for a particular Puppy.
This issue will be resolved when/if I can get compatible versions of these files into woof-ce 'testing', and then they can be removed from "overlay_init-1.0.tar".

Downloads:
for artfulpup-17.11+16 http://www.fishprogs.software/puppy/ini ... ul-0.9.tar.
for LxPupSc-18.04+3T-k64 http://www.fishprogs.software/puppy/ini ... sc-0.9.tar.
for slacko-700-RC3_k4.4.70-PAE_2018-04-05_d553a8c-patched http://www.fishprogs.software/puppy/ini ... ko-0.9.tar.
for slacko64-6.9.9.9-uefi_r6048 http://www.fishprogs.software/puppy/ini ... 64-0.9.tar.
for Upup_Bionic_Beaver-18.04+7 http://www.fishprogs.software/puppy/ini ... bb-0.9.tar.
for xenialpup-7.5-uefi http://www.fishprogs.software/puppy/ini ... al-0.9.tar.

Deltas:
for artfulpup-17.11+16.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.
for LxPupSc-18.04+3T-k64.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.
for slacko-700-RC3_k4.4.70-PAE_2018-04-05_d553a8c-patched.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.
for slacko64-6.9.9.9-uefi_r6048.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.
for Upup_Bionic_Beaver-18.04+7.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.
for xenialpup-7.5-uefi.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.

Changes:
1. Simplified the "Squash" facility in "Overlay init setup".
This used to consist of a "split" process on shutdown, and a "purge" process after the next boot.
The "split" process has been optimised so that the gain from a "purge" process is not worth the complication, so the "purge" process had been eliminated.
The "Squash" process used to produce an update by "rc.update", this is now skipped.
2. Re-versioned all the untilities to 0.9
3. Removed "/usr/sbin/bootmanager" from most versions because the modification is now part of woof-ce 'testing'.
4. Removed support for "pbootpart" and "pbootdir" boot parameters from "init" script.
5. Renamed "OVERLAY_SPECS" to "initrdBOOTspecs".
6. Fixed bug, multisession DVD not saving when using "Archive" save mechanism.

Usage:
1. Download a delta file for a Puppy iso file that you already have, and use it to produce an "overlay" iso file.
Do a manual frugal install using this "overlay" iso,
or
install it on a usb stick using 01micko's "install2usb", see "overlay_installers-0.9.sfs.gz" in a previous post.
or
if you don't require UEFI boot, use rcrsn51's ISObooter, but answer "n" in response to it's "splash screen" question.
2. Download one of the tar files into a fresh frugal install of the appropriate Puppy,
run "overlay-setup-frugal", see "overlay_installers-0.9.sfs.gz" in a previous post.

Review, what's so interesting about this package:

1. It uses "overlayfs" not "aufs" to build the union stack.
This is suppoosed to be good because "overlayfs" is in the kernel, and so has a future,
"aufs" is not, and is never likely to be, so some predict that it will die.
"overlayfs" is supposed to be efficient, since all reading and writing of files goes directly to the actual files.

2. It contains a new facility called "bootspecs", this consists of a file that creates a mechanism for the running Puppy to communicate with the "init" script.
This means that the "init" script can more truly be said to "only do what it's told".
This facility includes scripts that make it very easy for any utility in the running system, to communicate with "init", e.g. "initmodules".

3. The complicated boot parameters for specifying what system sfs files to load, are gone.
Instead, in the running Puppy, the "Overlay init setup" GUI utility is used to "slelect" the sfs files to load and their order in 2 system sfs lists.
The "ABOVE" list (defult adrv,ydrv) that gets mounted in the stack above the "MAIN" list (default puppy,fdrv,zdrv).
These lists can contain virtually any number of sfs files with any name and with any location.
The sfs lists facility relies on the "bootspecs" facility.

4. While there is support for "psavepart" (or it's alias "psave") and "psavedir" boot parameters, these are only very rarely needed.
The "saveconfig" utility, (accessed via "Overlay init setup") can be run at any time to "select" the save mode and "select" the location where the save is stored.
So you can copy your savefolder to a different location, or change your save mechanism from "Folder" to "Archive".
The "saveconfig" utility is run automatically on first shutdown.
On first shutdown, if your are not certain that you have a suitable Linux partition for your savefolder, simply choose the "Archive" save mechanism.
When you have sorted your Linux partition, run "Overlay init setup", "Saveconfig", choose "Folder" and "Select" a directory in your Linux partition.
Again, the relevant information is communicated to "init" via the "bootspecs" facility.
And when you choose a savefolder, you always get a proper pupmode=12, never a fiddly pupmode=13.

5. To avoid any chance that Puppy will boot a different copy of the same Puppy version,
the "psfspart" boot parameter (or one of it's aliases, "pdev1", "pupsfs", "pdrv") is required.
And if the Puppy sfs files are stored in a sub-directory of the defined partition, "psfsdir" (or it's alias "psubdir") is required.
Note, that it does not care where the "vmlinuz" and "initrd.gz" files reside, these are already loaded into memory before "init" starts.
The only exception is if you are booting an iso or DVD, i.e. with "pmedia=cd" in which case the "iso9660" partition is chosen.

6. The "saveconfig" utility supports the creation and use of a Luks encrypted partition to contain savefolders.
Instead of having a Luks savefile for a single Puppy, you have a real Luks partition, that can be used to contain the savefolders of many Puppies.

7. The "Archive" save mechanism provides a means of having your cake and eating it too,
i.e having Puppy running totally in RAM, but having changes saved on shutdown and restored on boot.
Good for a limited use Puppy on a cheap usb stick.

8. Multisession DVD support has been enhanced to include the facilities provided by "Overlay init setup", including the selection of sfs file.
So, it's possible to "upgrade" a multisession DVD by writing a replacement sfs file to it,
and then using "Overlay init setup" to select the new sfs file instead of the old sfs file in the appropriate list.
Or you can simply select sfs files located anywhere on a HD, or select a savefolder location on a HD.
The "Archive" save mechanism is implemented for a multisession DVD, each archive file is appended to the DVD at shutdown.

9. "Overlay init setup" also contains a facility to specify some of the "pfix=" boot flags.
This includes a "nosave" facility that works with all save mechanisms, including what we are used to calling "pupmode=12".

10. The "Extra-SFS manager" supports the "Extra" sfs file list, that gets loaded into the stack below the "MAIN" list.
Again, you can select almost any sfs file, to be included in this list.
Unfortunately, extra sfs files are not loaded immediately, they are loaded by "init" because with "overlayfs", the stack cannot be modified after it is mounted.

11. Replacing an sfs file with another of the same name, now leads to "rc.update" displaying the dreaded "next boot will be faster" message.
The information passed from "init" to "rc.update" has been enhanced to include a modification time stamp.
The updating mechanism in "rc.update" has also been enhanced to be a lot more selective to do only what the contents of the "changed" sfs file suggests is needed.

12. Data security is enhanced by "init" always preforming an fsck on a partition before it's first mount.
This is the default behaviour, it can be disabled by booting with "pfix=nofsckp".

13. The timezone that applies during "init" and the subsequent running system can now be specified in "initrd.gz".
When an "overlay_init" Puppy is installed to a usb stick via the "install2usb" script, this is done automatically.
It is also done automatically when "overlay-setup-frugal" is used to convert a "normal" Puppy to an "overlay" Puppy.

14. A read-only stack is mounted as "/initrd/pup_sfs", this contains all the sfs's.
So it's easy to determine if a file exists in any of the loaded sfs's, and if so, read it.

Limitations imposed by "overlayfs":

1. Layers can neither be added to, nor removed from, an "overlayfs" stack.
This means the "sfs_load" as we know it, cannot be supported.

2. The directories that makeup the "overlayfs" stack, cannot be modified directly.
This means there can be no "snapmergepuppy", nor can "PPM" install directly tp "pup_ro1".

3. A read/write "overlayfs" stack, requires 2 directories on the same filesystem for it's top layer.
I have implemented this requirement by always having a "work" directory beside the "save" directory.
This means we can't use a mount point as the top directory, it would have to be 2 directories inside the mount point directory.
So, a "savefile" mechanism could not be implemented in the same manner as current.
Also using a partition as a save layer is not possible.

A note on not "searching":

Whenever there is "searching", there is the chance that more than 1 file exists that matches the search critera,
and the "found" file, might not be the one expected.
There is an advantage in avoiding "searching" if at all possible.

When you install a Puppy, you must know exactly where you installed it.
So to avoid any chance of surprises, why not tell "init" exactly where you installed it.
This "init" insists that you do.

When you select where your changed files should be saved, this "saveconfig" tells "init" exactly where it is saved, via "bootspecs".

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Cleanup

#171 Post by gyro »

Over the next few days, I will cleanup the download site to remove all the old stuff, leaving just the version 0.9 downloads.
gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

upupbb-18.05.iso

#172 Post by gyro »

I have uploaded a delta for upupbb-18.05.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.

And re-uploaded http://www.fishprogs.software/puppy/ini ... bb-0.9.tar to match it.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#173 Post by gyro »

I have uploaded http://www.fishprogs.software/puppy/ini ... bb-0.2.pet for upupbb-18.05.
gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#174 Post by gyro »

I have attached "overlay_installers-0.9b.sfs".

There is a bug in "install2usb".
The generated "grub.cfg" file for a uefi install is missing a "pmedia=usbflash" in the first "linux" line.
This only makes a difference if you are installing a "normal" puppy as uefi, since "overlay_init" ignores this "pmedia" bootparameter.

gyro
Attachments
overlay_installers-0.9b.sfs.gz
gunzip to produce "overlay_installers-0.9b.sfs"
(9.87 KiB) Downloaded 444 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#175 Post by gyro »

Before anyone asks, yes, this bug exists in the "install2usb" included in overlay_init-0.9, it will be fixed in the next release, probably overlay_init-0.9.1.

Work around:
Simply edit the "grub.cfg" file on the usb stick and insert the missing "pmedia=usbflash"

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#176 Post by peebee »

Hi gyro

Any plans for updates for woof-ce-master builds?

I tried http://www.fishprogs.software/puppy/ini ... sc-0.9.tar with LxPupSc-18.06 but it was unable to find DISTRO_SPECS in the frugal directory no matter what combination of pdrv and psubdir I tried....

Thanks
peebee
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#177 Post by s243a »

peebee wrote:Hi gyro

Any plans for updates for woof-ce-master builds?

I tried http://www.fishprogs.software/puppy/ini ... sc-0.9.tar with LxPupSc-18.06 but it was unable to find DISTRO_SPECS in the frugal directory no matter what combination of pdrv and psubdir I tried....

Thanks
peebee
I thought that the OverlayFs didn't support all the pupmodes so it should probably be a boot option rather than the only option.

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

overlay_init-0.9b.tar

#178 Post by gyro »

I have uploaded http://www.fishprogs.software/puppy/ini ... t-0.9b.tar for curent versions of LxPupSc and upupbb.
This is to reflect recent changes to woof-ce that obselete some of my patches.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#179 Post by gyro »

peebee wrote:Any plans for updates for woof-ce-master builds?
Yes, but only to patch some exsiting utilities to be compatible with both "normal" and "overlay" puppies.
peebee wrote:I tried http://www.fishprogs.software/puppy/ini ... sc-0.9.tar with LxPupSc-18.06 but it was unable to find DISTRO_SPECS in the frugal directory no matter what combination of pdrv and psubdir I tried....
What process were you running that failed to find DISTRO_SPECS?
The current "overlay-setup-frugal" uses the "DISTRO_SPECS" from the current "initrd.gz", it does not need "DISTRO_SPECS" in the frugal directory.
I did a manual frugal install of LxPupSc-18.06+1M-k64. (vmlinuz, initrd.gz, and all the .sfs files.)
Copied "overlay_init-0.9b.tar" (see previous post) to this frugal directory.
Then ran "overlay-setup-frugal". It ran to completion.
A re-boot using the resultant frugal directory was sucessfull.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#180 Post by peebee »

gyro wrote:What process were you running that failed to find DISTRO_SPECS?
The current "overlay-setup-frugal" uses the "DISTRO_SPECS" from the current "initrd.gz", it does not need "DISTRO_SPECS" in the frugal directory.
I did a manual frugal install of LxPupSc-18.06+1M-k64. (vmlinuz, initrd.gz, and all the .sfs files.)
Copied "overlay_init-0.9b.tar" (see previous post) to this frugal directory.
Then ran "overlay-setup-frugal". It ran to completion.
A re-boot using the resultant frugal directory was sucessfull.

gyro
Hi gyro

I was following the instructions in post #1 which makes no mention of "overlay-setup-frugal".....and overlay_setup_frugal does not seem to be in "overlay_init-0.9b.tar" which has just 2 components - initrd.overlay.gz and overlay_mods.sfs

Can you expand the instructions in post #1 to say how to find and run "overlay-setup-frugal"?

I see that "/usr/sbin/overlay-setup-frugal" is in overlay_mods.sfs but do I install the .sfs and then run it or do I extract it and run it??

Thanks
peebee
post#1 wrote:To use:
In a clean frugal install:
1. Add a copy of DISTRO_SPECS to the frugal install directory.
2. Download the "initrd.gz" and "ydrv_overlay.sfs" into the frugal install directory.
3. Boot, with boot parameters similar to the following:
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

Post Reply