Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Message
Author
User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

Re: 01micko's install2usb

#141 Post by 01micko »

gyro wrote:Note: If this script were to be aimed at only "overlay_init" then it would be preferable to use "psfspart=" and "psfsdir=" instead of "pupsfs=" and "psubdir=".
We'll see where this goes first. May well be worth including 'overlay' facility in mainline 'puppy'.. or not.
gyro wrote:Additional features I would like it to have:
1. An option to have a normal ext4 partition, i.e. with journal.
Given the price of USB sticks, I think the "cost" of the journal is good value as insurance against data corruption.
2. An option to specify a label for the Linux partition.
The label on the vfat partition could also be optional and able to be specified.

gyro
Done and done. See revision 4.
Puppy Linux Blog - contact me for access

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: 01micko's install2usb

#142 Post by gyro »

01micko wrote:Done and done. See revision 4.
Thanks. I'll test.
I intend to include this installer in my next release of "onverly_init", so I've been playing with it a bit and come up with a couple of other possible changes.
1. Include a copy of "puppy.png" with the binary, and always use it.
(I assume that the host Puppy and the target Puppy will be different.)
2. Make "ext4 (no journal), the default with "ext2" last.
3. Align the partitions to 4MiB.
(I came across a suggestion that some usb sticks work better this way, although it's not been obvious in my testing.)
4. Copy "vmlinuz" and "initrd.gz" into a sub-directory of the fat32 partition with the same name as the sub-directory on the Linux partition.
This means a second puppy can be added manually, using this one as a template.

Sorry about this, I'll get back with another "patch" for these suggestions, after I've tested your new version.

I've been tesing on an el-chepo 8MiB stick, thats very slow at writing,
"ext2" was so slow, I almost decided that the machine was hung.
"f2fs" worked fine with a UEFI boot, but failed to boot with SYSLINUX. This may be a problem with the SYSLINUX on my host Puppy.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Recent kernel 4.9.86

#143 Post by gyro »

I've uploaded "huge-4.9.86-gyro-pae.tar.bz2". This kernel was released on 2018-03-03.

I compiled it on Xenialpup 7.5, using the woof-ce kernel-kit,

It's a 32bit PAE kernel.
It contains only a small set of firmware.
It contains overlayfs support built-in.

Downloads:
http://www.fishprogs.software/puppy/ini ... ae.tar.bz2
http://www.fishprogs.software/puppy/ini ... z2.md5.txt
http://www.fishprogs.software/puppy/ini ... sha256.txt

I use this kernel for all my xenialpup 7.5 Puppies.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#144 Post by gyro »

01micko wrote:* bugfix: copy appropriate BOOT_SPECS to initrd,gz
* feature: add P_TZ and P_TZONE vars to TIME_ZONE file (based on running systems timezone) >> initrd.gz
Unfortunately both these are obseleted by the next version of "overlay_init". In that neither of these files now exist in "initrd.gz".

I'm not sure that a general installer should be updrading the "initrd.gz", rather it should just be installing the Puppy that's provided.

On the other hand I like the idea of adding timezone information.
Wheras it's inappropriate to have timezone information in an initrd.gz in an iso,
it's quite resonable to have timezone information in the initrd.gz of a local install.

So I'll bring forward the next relsease of "overlay_init" so this installer can be properly sorted before the final release.
For this version I'll include a slightly modified version of your version 4 installer, as an application. It will update the timezone information in initrd.gz if it's an 'overlay' Puppy and there is none already there.

I've tested your version 4, with 2 modifications:
* changed the order of the filesystems so that "ext4 (no journal)" is the first, and therefore default.
* deleted the 2 obseleted features as per above.
Both UEFI and LEGACY boots worked fine using "ext4 (no journal)".
Also the fsck issue with the vfat partition is "fixed", though it looks a bit scary when the installer is run from a terminal.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: 01micko's install2usb

#145 Post by gyro »

gyro wrote:I intend to include this installer in my next release of "onverly_init", so I've been playing with it a bit and come up with a couple of other possible changes.
1. Include a copy of "puppy.png" with the binary, and always use it.
(I assume that the host Puppy and the target Puppy will be different.)
2. Make "ext4 (no journal), the default with "ext2" last.
3. Align the partitions to 4MiB.
(I came across a suggestion that some usb sticks work better this way, although it's not been obvious in my testing.)
4. Copy "vmlinuz" and "initrd.gz" into a sub-directory of the fat32 partition with the same name as the sub-directory on the Linux partition.
This means a second puppy can be added manually, using this one as a template.
On further reflection, only 2 is important.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Release of overlay_init-0.5

#146 Post by gyro »

Release of overlay_init-0.5

This release is to get the latest (and hopefully the last) version of "init" out there, so that others don't waste their time doing stuff that I am busily obseleting.

The utilities are now pretty much functionally complete, but may get some re-organisation.

Download from http://www.fishprogs.software/puppy/ini ... it-0.5.tar.

There's a delta for artfulpup-17.11+15.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.
And a delta for slacko-700-RC3_k4.4.70-PAE_2017-12-10.iso http://www.fishprogs.software/puppy/ini ... .iso.delta.

Changes for this version:
1. Instead of "initrd.gz" containing a TIME_ZONE file and a default bootspecs file, it now contains just an "OVERLAY_SPECS" file.
The bootspecs file exists only outside "initrd.gz" and is now called "${DISTRO_FILE_PREFIX}BOOTspecs".
2. Contains a version of 01micko's "install2usb" as "USB Installer - Flash Drives" in the "Setup" menu.
I've modified it a bit so I've changed the version to "0.4.1", my patch is attached.
The patch, changes the order of the filesystems in the selection dialog.
It also replaces the obseleted "initrd.gz" functions with a timezone facility that works with the current "init".

gyro
Attachments
install2usb-0.4.1.diff.gz
gunzip to produce "install2usb-0.4.1.diff"
(1.09 KiB) Downloaded 231 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

comments on "install2usb"

#147 Post by gyro »

"install2usb" is great, it's rather cool to be able to install an "overlay" iso to a usb stick and have the timezone automatically set.
So again, a big thanks to 01micko for this utility.

I recommend that labels are used to identify the partitions(s), just make sure that they are unique for your computer.

The layout of files over the 2 partitions is interesting, i.e. "vmlinuz" and "initrd.gz" on the vfat boot partition and the puppy sfs files on the Linux partition.

This overlay "init" has no problem with this, but there could be some possible confusion when changing the kernel, with the "vmlinuz" in one partition and the "zdrv" in the other. It may be an idea to provide some mechanism to indicate to the booted Puppy where the "vmlinuz" file is located.

The 'testing' "init" has no problem with this, but the subsequent running system does, i.e. "initmodules -q" does nothing and "bootmanager" does not show "initmodules" as an option.
In both cases it's because they can't locate "initrd.gz" in the frugal install directory with "puppy...sfs".
This "overlay_init" package contains patches so that both "initmodules" and "bootmanager" look for the "DOIMODS" variable in "/etc/rc.d/PUPSTATE" instead of greping "initrd.gz" for "pimod".
This patch could be ported to woof-ce 'testing'.
This patch was introduced because it's a lot lot faster starting "intmodules" or "bootmanager" off a slow usb stick.

The "classic" "init" script would not handle this at all, because it searches for an appropriate "vmlinuz" and assumes that that is the frugal install directoty.

gyro

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

Re: comments on "install2usb"

#148 Post by 01micko »

gyro wrote:"install2usb" is great, it's rather cool to be able to install an "overlay" iso to a usb stick and have the timezone automatically set.
So again, a big thanks to 01micko for this utility.
You're welcome. :)
gyro wrote:The layout of files over the 2 partitions is interesting, i.e. "vmlinuz" and "initrd.gz" on the vfat boot partition and the puppy sfs files on the Linux partition.
There are 2 reasons for this. Probably the most important is that UEFI requires a FAT filesystem to do it's thing.The kernel and initial ramdisk must be on the same partition as the UEFI stuff. The second is that at this time, no bootloader supports F2FS filesystem. I expect the latter to change if F2FS ever grows in popularity but I doubt that the former will at any point in time soon. Why such a modern technology as EFI decided that using FAT to boot was a good idea is way beyond me, but I suppose there are reasons. Perhaps the simplicity and the universal nature of it had it's appeal to the developers.
gyro wrote:This overlay "init" has no problem with this, but there could be some possible confusion when changing the kernel, with the "vmlinuz" in one partition and the "zdrv" in the other. It may be an idea to provide some mechanism to indicate to the booted Puppy where the "vmlinuz" file is located.
You are correct and I am aware of the issue and I suppose I should get this sorted before your final overlay release. 'testing' also has this bug because the F2FS usb installer naturally does the same thing (boots off fat, runs on linux FS partition)
gyro wrote:The 'testing' "init" has no problem with this, but the subsequent running system does, i.e. "initmodules -q" does nothing and "bootmanager" does not show "initmodules" as an option.
In both cases it's because they can't locate "initrd.gz" in the frugal install directory with "puppy...sfs".
This "overlay_init" package contains patches so that both "initmodules" and "bootmanager" look for the "DOIMODS" variable in "/etc/rc.d/PUPSTATE" instead of greping "initrd.gz" for "pimod".
This patch could be ported to woof-ce 'testing'.
This patch was introduced because it's a lot lot faster starting "intmodules" or "bootmanager" off a slow usb stick.

The "classic" "init" script would not handle this at all, because it searches for an appropriate "vmlinuz" and assumes that that is the frugal install directoty.
yes the patch probably should be ported to 'testing'. I should test and fix the 'install2usb' script for testing which will make obsolete the UEFI and F2FS usb installers. I'll get onto that after I fix the 'change_kernels' issue.

I have been thinking that if a users wants an ext(2,3,4) save partition and doesn't want UEFI (or can't use UEFI) to make a single partition with 'install2usb'. It does make things simpler in those cases. I'll think about that.

BTW, v0.5 of overlay init is working ok for me after some initial upgrade teething problems from 0.4 on my slacko64 frugal install to my laptop. I intend to burn a new dvd of the 32 bit slacko v0.5 (updated with delta) and perform a usb install off the dvd. Soon.

Cheers and a big thanks to you too for the overlay init initiative!
Puppy Linux Blog - contact me for access

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Re: comments on "install2usb"

#149 Post by gyro »

01micko wrote:
gyro wrote:This overlay "init" has no problem with this, but there could be some possible confusion when changing the kernel, with the "vmlinuz" in one partition and the "zdrv" in the other. It may be an idea to provide some mechanism to indicate to the booted Puppy where the "vmlinuz" file is located.
You are correct and I am aware of the issue and I suppose I should get this sorted before your final overlay release. 'testing' also has this bug because the F2FS usb installer naturally does the same thing (boots off fat, runs on linux FS partition)
My first thought would be for the kernel change script to "guess" that the "vmlinuz" file is in the root of the "1" partition corresponding to the "PDEV1" partition.
Or write a text file, e.g. "vmlinuz_location", in the sfs directory on the Linux partition that contains information that allows the kernel change script to establish the correct location of "vmlinuz".
Just my 2 cents worth.
01micko wrote:
gyro wrote:This "overlay_init" package contains patches so that both "initmodules" and "bootmanager" look for the "DOIMODS" variable in "/etc/rc.d/PUPSTATE" instead of greping "initrd.gz" for "pimod".
This patch could be ported to woof-ce 'testing'.
This patch was introduced because it's a lot lot faster starting "intmodules" or "bootmanager" off a slow usb stick.
yes the patch probably should be ported to 'testing'.
I'll try to do this soon.
I have a little git problem to sort first.

Later edit:
Support for "DOIMODS" is now committed in woof-CE 'testing'.

gyro

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

change_kernels

#150 Post by 01micko »

Attached is a diff for the /usr/sbin/change_kernels script taken from slacko-overlay0.5-700-RC3_k4.4.70-PAE_2017-12-10.iso (created from delta).

While I conclude that it works with a split fat/ext (boot/save on separate partitions - typical of the 'install2usb' script) I did have a failure. Using the huge-4.9.86-gyro-pae.tar.bz2 from here the kernel change appeared to go fine but on reboot my system would not boot. Probably not the fault of the adjusted script but an incompatibility with slacko versus xenial; something to do with ext4. In the end the main sfs could not be found and I was dropped to the init prompt.
Later I changed kernels with my slacko64 kernel (because this hardware is 64 bit capable) and the change of kernel and boot went fine.

You can of course use the attached diff with the 'patch' utility after it is decompressed. This isn't tested with the 'testing' init as yet but should be OK. Will test later,
Attachments
change_kernels.diff.gz
(701 Bytes) Downloaded 238 times
Screenshot.png
(52.56 KiB) Downloaded 340 times
Puppy Linux Blog - contact me for access

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#151 Post by gyro »

@01micko,

I just tested a patched "change_kernels" script and it worked fine.

I setup a directory containig the files from the release xenialpup 7.5 iso.
I then patched it's "change_kernels" script with your diff and added it as a ydrv.
I then installed these files to a usb stick using "install2usb".
After rebooting the usb stick into pupmode=13, I ran the patched "change_kernels" script, to upgrade to my 32bit PAE 4.9.86 kernel.
On reboot everything worked fine, using the new kernel.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#152 Post by gyro »

@01micko,
I have a slight concern with your patch to "change_kernels":
Could it cope with a device that contains multiple Puppies, each in their own PSUBDIR, but with files split between sdx1 and sdx2 a bit like "install2usb" except that "vmlinuz" and "initrd.gz" are in their own PSUBDIR on sdx1, (so sdx1 can cope with multiple Puppies)?

I'm assuming that this "install2usb' will become the "standard" installer for usb sticks.
And then some folk might use it's file layout as a template to setup a multiple Puppy usb HD.

PS: This instller should be seen as the upcomming "standard" installer for usb sticks, and have it's own forum topic.

gyo

User avatar
01micko
Posts: 8741
Joined: Sat 11 Oct 2008, 13:39
Location: qld
Contact:

#153 Post by 01micko »

gyro wrote:@01micko,
I have a slight concern with your patch to "change_kernels":
Could it cope with a device that contains multiple Puppies, each in their own PSUBDIR, but with files split between sdx1 and sdx2 a bit like "install2usb" except that "vmlinuz" and "initrd.gz" are in their own PSUBDIR on sdx1, (so sdx1 can cope with multiple Puppies)?
Not multiple. It is only searching for 'vmlinuz'. However, as you previously mentioned, it would be ideal to place the boot files (vmlinuz, initrd.gz) in a PSUBDIR at the root of the vfat partition. That way, the 'change_kernels' script could search for $PSUBDIR/vmlinuz. Much neater if one does want to have multiple installs to a USB device.
gyro wrote:I'm assuming that this "install2usb' will become the "standard" installer for usb sticks.
And then some folk might use it's file layout as a template to setup a multiple Puppy usb HD.

PS: This instller should be seen as the upcomming "standard" installer for usb sticks, and have it's own forum topic.

gyo
Well there is no reason why it shouldn't be the 'standard', but I think I would still like to include single partition installs for ext[2,3,4] when UEFI is not wanted/not an option. ALL files should go to the PSUBDIR in their respective partitions.

I'll tidy that up next and then create a thread for 'install2usb' (which may have a name change.. something like 'pupinstall2usb' :? ? - open to suggestions)
Puppy Linux Blog - contact me for access

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#154 Post by gyro »

@01micko,

I have another suggestion concerning the "hidden" "vmlinuz" issue.
When running "install2usb" on the host Puppy, the location of "vmlinuz" is known. So how can we communicate this information to "change_kernels" in the runnng target Puppy?

In "install2usb", a "pvmlinuzpart" boot parameter is added for the target Puppy, whose value is the LABEL or UUID of the vfat partition.
The "init" script is modified so that it picks-up "pvmlinuzpart" and writes it to PUPSTATE as "PVMLINUZPART", (also add support for "pvmlinuzdir" for completeness). This is a fairly trivial patch to "init", so it could be applied to any "init" script.
In "change_kernels", if "PVMLINUZPART" is empty then use "PDEV1".

It looks complicated, but none of the modifications are difficult, and the result is very flexible, and it's all setup by the process that "knows".
"change_kernels" doesn't have to "guess".

I'll upload an "overlay_init-0.5b.tar" containg a patched "init" script.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#155 Post by gyro »

01micko wrote:I'll tidy that up next and then create a thread for 'install2usb' (which may have a name change.. something like 'pupinstall2usb' :? ? - open to suggestions)
How about 'pupinstall2usbstick'.
Unless it is also targeted to support multiple frugal installs to a usb HD, i.e. an option to skip partitioning.
Then 'pupinstall2usb'.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Release of overlay_init-0.5b

#156 Post by gyro »

This is a testing version with a modified "init" script to support 2 new boot parameters, "pbootpart" and "pbootdir".
The "init" scipt doesn't actually use them, it just publishes them in the PUPSTATE file as "PBOOTPART" and "PBOOTDIR", although it does decode "PBOOTPART" from a label or uuid into a partition name.
They are meant to support situations where the "vmlinuz" and "initrd.gz" files are in a different partition from the "puppy...sfs", like a "install2usb" install.

The idea is that an installer like "install2usb" could add a "pbootpart" boot parameter that specifies the label or uuid of the vfat partition.
Then a script in the running target Puppy, like"change_kernels", can pickup the partition name corresponding to this value from "PBOOTPART" in the PUPSTATE file.

Please note the name change from my previous post.

Download from http://www.fishprogs.software/puppy/ini ... t-0.5b.tar.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

01micko's install2usb - latest hack

#157 Post by gyro »

I have attached "overlay_installers-0.6.sfs".
This sfs is meant to be loaded as an extra sfs in a "normal" Puppy.
It contains my latest hack of "install2usb" which will install both "normal" and "overlay" Puppies.
It should appear in the "Setup" menu as "USB Installer - Flash Drives".
When run via this sfs, it will run on "normal" Puppies.

The sfs also contains "overlay-setup-frugal", which will run on "normal" Puppies via this sfs.

I have also attached "change_kernels".
This has been patched to support "PBOOTPART" and "PBOOTDIR', but will work without them.
It shoud run on both "normal" and "overlay" Puppies.

I have not included "change_kernels" in the sfs becasue being in an extra sfs, it would be "covered" by the installed "change_kernels".

Note on "pbootpart" and "pbootdir" boot parameters:
This "install2usb" always specifies them so they can be used if the "init" script in the target Puppy supports them.
If the "init" script in the target Puppy does not support them, they will simply be ignored.

gyro
Attachments
change_kernels.gz
gunzip to produce "change_kernels"
(4.38 KiB) Downloaded 254 times
overlay_installers-0.6.sfs.gz
gunzip to produce "overlay_installers-0.6.sfs"
(10.02 KiB) Downloaded 260 times

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Release of overlay_init-0.6

#158 Post by gyro »

Release of overlay_init-0.6

This release is to fully support my latest hack of "install2usb".

Download from http://www.fishprogs.software/puppy/ini ... it-0.6.tar.

There are also deltas for:
artfulpup-17.11+16.iso http://www.fishprogs.software/puppy/ini ... .iso.delta
LxPupSc-18.04T-k64.iso http://www.fishprogs.software/puppy/ini ... .iso.delta
slacko64-6.9.9.9-uefi_r6048.iso http://www.fishprogs.software/puppy/ini ... .iso.delta
xenialpup-7.5-uefi.iso http://www.fishprogs.software/puppy/ini ... .iso.delta
Note: The delta's are mostly about 2MiB, but the xenialpup one is about 37MiB beacuse it contains an updated kernel (4.9.92) and zdrv, as well.

Changes for this version:
1. Includes my latest hacks of "install2usb" and "change_kernels" scripts.
2. "Overlay init setup" utility now includes a "Squash" button if you are running in pupmode=21, and "pfix=nocopy" is not set.
On the following shutdown, the savefolder will be split into an sfs file and a minimial archive file.
Subsequent boots will load the the sfs file at the top of the stack.
After the next boot, the the sfs file will be trimmed of any files added to the savefolder by the boot.
Subsequent shutdowns will revert to the normal "archive" process.
The idea is that this can be used when the archive file is becomming large, containing many files that don't change, e.g. after installing a few pets.
3. The "init" script now aborts if it can't find any kernel modules after loading the "MAIN" sfs list.

Using:
The preferred way is to generate an iso from one of the provided delta files.
Then use "install2usb" (see "overlay_installers-0.6.sfs" in post above), to install to a usb stick.
Or do a manual frugal install from the iso.

Using an iso is preferred because "/usr/local/pup_event" has been changing, and the "/usr/local/pup_event/frontend_funcs" in "overlay_init-0.6.tar" is a patched version from xenialpup 7.5,
whereas the iso files each contain a patched version of the "/usr/local/pup_event/frontend_funcs" from their parent iso.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#159 Post by gyro »

I've uploaded new versions of:
LxPupSc-18.04T-k64.iso http://www.fishprogs.software/puppy/ini ... .iso.delta
slacko64-6.9.9.9-uefi_r6048.iso http://www.fishprogs.software/puppy/ini ... .iso.delta

They now contain the appropriate "rc.update".

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

patch files for "install2usb" and "change_kernels"

#160 Post by gyro »

I should have done this sooner.
I have attached "install2usb.diff" and "change_kernels.diff".
These are the hacks I have done to produce the utilities as released above.
Except that in the "install2usb.diff" the version is changed to "0.4.2" wheras the released code still has version "0.4.1".

gyro
Attachments
install2usb.diff.gz
gunzip to produce install2usb.zip
(2.23 KiB) Downloaded 235 times
change_kernels.diff.gz
gunzip to produce change_kernels.diff
(1.84 KiB) Downloaded 246 times
Last edited by gyro on Sun 15 Apr 2018, 01:46, edited 1 time in total.

Post Reply