Another re-write of the "init" script - using OverlayFs?

Under development: PCMCIA, wireless, etc.
Post Reply
Message
Author
gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Another re-write of the "init" script - using OverlayFs?

#1 Post by gyro »

Any remaining files for this project have been moved to http://www.mediafire.com/folder/inbkdjmie89hm/overlay. Please download from there.
If the file you want to download has not been transfered, please report it in this topic http://www.murga-linux.com/puppy/viewtopic.php?t=115959.
In the next few weeks most URL's beginning with http://www.fishprogs.software/puppy/* will fail.

Edit:
The release of overlay_init-0.9.
This version contains a number of download variants, please see this post http://www.murga-linux.com/puppy/viewto ... 777#989777 for details.

The easiest way to use overlay_init, is to create an iso using a released delta.
Download one of those available from http://www.mediafire.com/folder/inbkdjmie89hm/overlay.

But for a Puppy that has no released overlay delta, you can use a "normal" Puppy with a working sfs_load to modify a frugal install:

1. Do a fresh frugal install of your target Puppy into a suitable writeable directory.
Make sure the "kernel" line for this install in your boot loader config contains
at least an appropriate "pupsfs=<partition>" parameter and a "psubdir=<sub-directory>" parameter if required.
Where "<partition>" is the LABEL or UUID of the target partition, and "<sub-directory>" is the path to the target directory.

2. Download an overlay_init .tar file from http://www.mediafire.com/folder/inbkdjmie89hm/overlay e.g. overlay_init-0.9b.tar to the new install directory.
Open a console window in the new install directory, and run command "tar xf overlay_init-0.9b.tar".
This should produce 2 files, "initrd.overlay.gz" and "overlay_mods.sfs".

3. Load "overlay_mods.sfs" using sfs_load.
This will make a number of utilities specific to this 'init', available to the running Puppy.

4. Run the command "overlay-setup-frugal <install-directory>", where "<install-directory>" is the full path to your install directory.

5. Unload "overlay_mods.sfs" using sfs_load.

6. Boot this modified installation.

Instead of loading "overlay_mods.sfs" with sfs_load, you could load "overlay_installers-0.10.sfs" from http://www.mediafire.com/folder/inbkdjmie89hm/overlay
See this post, http://www.murga-linux.com/puppy/viewto ... 083#988083, for more detail.

Edit:
---Old stuff, now obselete, the way it started---

Replacements for standard puppy files:
http://www.fishprogs.software/puppy/initrd/init
http://www.fishprogs.software/puppy/initrd/initrd.gz
http://www.fishprogs.software/puppy/ini ... verlay.sfs

Possibly usefuly utilities:
http://www.fishprogs.software/puppy/initrd/file2initrd
http://www.fishprogs.software/puppy/initrd/initrd2file

An example text file:
http://www.fishprogs.software/puppy/initrd/TIME_ZONE

To use:
In a clean frugal install:
1. Add a copy of DISTRO_SPECS to the frugal install directory.
2. Download the "initrd.gz" and "ydrv_overlay.sfs" into the frugal install directory.
3. Boot, with boot parameters similar to the following:

Code: Select all

pdrv=sdc2:/slacko/ ydrv=:ydrv_overlay.sfs
Note: If the kernel in your puppy does not support "overlayfs", "init" will fail.

---Original message---
I'm not sure if this is a warning, a threat or a promise.
I'm working on another rewrite of the "init" script.

Amongst other things, I want to explore some different technologies, one of these is OverlayFs.
Some folk are saying that aufs is dead, that OverlayFs is the way of the future because it is included in the kernel.
This prediction may or may not happen, but I am interested in finding out what it would mean for puppy it we moved to OverlayFs as a replacement for aufs.

The results of my reading research are:

1) OverlayFs is much simpler that aufs, it seems to lack any ability to modify an existing stack.
If this proves to be true in practice, then the stack would have to be created once in "init", and then remain unchanged for the session.
"sfs_load" would be reduced to being an sfs manager whose results did not take effect until after a reboot.
Utilities that execute sfs files, by loading them into the aufs stack, executing the program inside and then unloading them would go.
There might also be some challanges with pupmode=13, since what happens when a directory that is part of an overlay is modified directly is undefined. So "snapmergepuppy" might have to go.

2) There are similarities, a number of directories are combined together to form a stack,
sfs files have to be mounted and their mountpoints used in forming the stack,
it uses whiteout files in the rw layer to hide files in the ro layers that have been "deleted".

3) But it can have read only overlays, and an overlay can consist of more than 1 directory, which could themselves be overlays.
So we could create a puppy-sfs overlay(ro) containing all the puppy system sfs's, and an extra-sfs overlay(ro) and a save-layer overlay(rw) and then mount all these as a puppy overlay.
Or we could build an sfs overlay(ro) that contains all the sfs's in the current session, and combine that with a save-layer directory to produce the puppy overlay. One of the advantages of this approach is that it would be easy to find if a file exists in any of the sfs's, by simply testing it's existence in the directory where the sfs overlay is mounted.
Or we could build an sfs overlay gradually by combining the previous sfs overlay with the directory for this sfs file to produce a new overlay, and so on. The challange with this approach is that each new overlay needs a new directory.

The next thing to do is to try using OverlayFs, so my questions are:
Has anyone tried to use OverlayFs?
Since it is a kernel facility, is it enabled in current Puppy kernels?
Or do I need do a kernel compile before I can try using it?

gyro
Last edited by gyro on Tue 23 Apr 2019, 20:38, edited 17 times in total.

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#2 Post by peebee »

BarryK has posted some blogs about use of overlayfs - e.g.
http://barryk.org/news/?viewDetailed=00499 - search by content for others....

In my /etc/modules/DOTconfig-4.11.2-210517 it says:
# CONFIG_OVERLAY_FS is not set
so I would imagine it would need to be set and the kernel re-built.....

Cheers
peebee
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#3 Post by gyro »

@peebee,
Thanks.

Unfortunately BK confirms the limitations of overlayfs.
Although I think a pupmode=12 should be doable.
A pupmode=5 might be a bit more difficult, might have to replace tmpfs with a zram.
A pupmode=13 would be the biggest challange, probably have to create a new save layer (pup_ro1) on shutdown and have "init" use the new one and delete the old one.
So still some possibilities that could be explored.

Looks like a kernel compile is needed if I want to try this stuff.

gyro

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#4 Post by BarryK »

I am using overlay f.s. for Quirky and Easy. Though, Quirky is a full installation and doesn't need it, except it is used in woofQ, in the '3builddistro' script -- and I presume, still, also in woofCE.

Yes, only the top layer is rw, all layers below are ro, and cannot be written to at all. So, it does pose a challenge for some of those pupmodes.

In Easy, I am using just one mode, the rw layer is a folder in a drive, requiring a partition with Linux f.s. So there is no need for a save-file, nor a snapmergepuppy.
This is a restriction, cannot use a fat or ntfs partition.
[url]https://bkhome.org/news/[/url]

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#5 Post by rufwoof »

To upgrade temporarily from Debian Jessie frugal (aufs) to Stretch (overlayfs) I had to install aufs.dkms and add a kernel boot parameter of union=aufs After that it frugally booted/ran fine and sfs's can be loaded as normal. Not as ingrained as using in-built overlayfs, but looks to me that you might have to pair overlayfs with btrfs which is still somewhat evolving (less stable).

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#6 Post by gyro »

@BarryK,
Thanks for confirming that you have working, somethig that is effectively a pupmode=12 with a savefolder.
We'll see what happens to the rest.

@rufwoof,
Sorry, I don't understand your comment.
I'm not aware of any woof-ce based puppies that use overlays.
And I don't get what suggests that overlays work better on btrfs.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#7 Post by gyro »

I guess what I would really like to know is, any information on how to modify an overlay stack, if it's possible?

gyro

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#8 Post by rufwoof »

gyro wrote:@rufwoof,
Sorry, I don't understand your comment.
I'm not aware of any woof-ce based puppies that use overlays.
And I don't get what suggests that overlays work better on btrfs.
I thought the thread was to address a new init (puppy) that was based on using overlayfs.

With btrfs you can set things up as though one set of files/folders were the entire partition/volume, but where another set of files/folders were also as though in their own unique volume. i.e. one linux in one subvolume, another in another subvolume, where snapshots and 'backintime' type instances can be managed/used.

Not sure, but I think Barry's easylinux might be using btrfs ???

Haven't really thought it through, but to modify a overlayfs under btrfs it might be possible to set up a overlay that uses a particular subvolume, and then perhaps change that out for another choice of subvolume ... as though the overlayfs had been modified.

I believe a overlayfs can include overlays. Such that a existing overlayfs that was transitioned over to be one of the (read only) source of another overlays might be managed so that it seemed that the currently used (rw) overlayfs had been modified/extended so-to-speak. Combined with COW (copy on write) you might even have copies/versions of changes being applied to otherwise read-only layers if so needed.

The negative side is that there are rumours of a data-eating bug in some arrangements i.e. its perhaps not mature enough yet for use in anything other than relatively straight forward setups, and where using the likes of the above is more inclined to hit such data corruption problems.

Rangan Masti
Posts: 37
Joined: Tue 01 Jan 2013, 19:23
Location: Germany, Berlin

#9 Post by Rangan Masti »

I am interested in this and waiting egerly the init. Wish you good success!!!

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#10 Post by BarryK »

rufwoof wrote:Not sure, but I think Barry's easylinux might be using btrfs ???
No. The f.s. is ext4.
[url]https://bkhome.org/news/[/url]

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#11 Post by gyro »

I intend to test using ext4, my overlays will consist of simply stacking directories.

I now have a little test bed "init" that supports only pupmode=5, but no overlays yet, still using aufs.
I intend to test a save layer mechanism based on an sfs, first.
Then try doing it again with overlays.

I haven't even got a puppy running that is capable of overlays, yet.

I should be clear, the use of overlays is only one of the new things I'm trying with the new init.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#12 Post by gyro »

Just in case anyone is interested, my current test "init" can be downloaded from http://www.fishprogs.software/puppy/initrd/init.
There is also an "initrd.gz" at http://www.fishprogs.software/puppy/initrd/initrd.gz.

The "initrd.gz" does not contain any DISTRO_SPECS, so it can boot many recent woof-ce based puppies.
To test on an existing frugal install, (a testing one, not your main work one).
1. In the frugal install directory, rename the current "initrd.gz" to something else, (as backup).
2. Download the test "initrd.gz" and move it into the frugal install directory.
3. Copy "etc/DISTRO_SPECS" from inside your savefolder into the install directory, (beside "initrd.gz")
4. reboot.

To return to normal, just delete the new "initrd.gz" and rename your original one back to "initrd.gz"

Note1: This is a simple pupmode=5 boot, it will ignore your savefolder. So, don't bother setting up a save on first shutdown, just click on "No".

Note2: It does no searching at all, so you must tell it where the frugal install directory is with boot parameters, e.g. "pdrv=Work:/puppy/slacko/", where "Work" is the label on the ext4 partition that contains my frugal install directory ("/puppy/slacko/").

gyro

anikin
Posts: 994
Joined: Thu 10 May 2012, 06:16

#13 Post by anikin »

gyro wrote:...
The next thing to do is to try using OverlayFs, so my questions are:
Has anyone tried to use OverlayFs?
Since it is a kernel facility, is it enabled in current Puppy kernels?
Or do I need do a kernel compile before I can try using it?

gyro
Just have a look at your kernel dotconfig. For example, my kernel has both aufs and overlayfs enabled like this:

Code: Select all

CONFIG_AUFS_FS=m
CONFIG_OVERLAY_FS=m
Now you can choose between the two by adding either

Code: Select all

union=aufs
or

Code: Select all

union=overlay
as a kernel boot parameter.

I have some limited experience using a Debian Live system with overlayfs. As I'm not that much interested in its stacking or overlay (save) capabilities, it runs OK for me. However, it also has a huge drawback - an extremely slow and weird boot process.

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#14 Post by peebee »

Followed the instructions in post #956788 including putting DISTRO_SPECS into the frugal install folder - but did not boot - error message that DISTRO_SPECS could not be found.....

menu.lst entry is

Code: Select all

title Devtest (sda3/devtest)
  uuid 95a633e7-6fab-431d-9fa8-550812ab03f8
  kernel /devtest/vmlinuz  pmedia=atahd psubdir=devtest pfix=fsck
  initrd /devtest/initrd.gz
What am I doing wrong?? Does DISTRO_SPECS have to be inside folder /etc???
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#15 Post by Keef »

You need something like

Code: Select all

pdrv=sda2:/tahr/


Also (when using Tahr), the zdrv needs finding (if you need it for anything)

Code: Select all

zdrv=sda2:/tahr/zdrv_tahr_6.0.6.sfs

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#16 Post by peebee »

[FIXED] - see below

Tried both:

Code: Select all

kernel /devtest/vmlinuz  pmedia=atahd pdrv=sda3:/devtest/ psubdir=devtest pfix=fsck
and

Code: Select all

kernel /devtest/vmlinuz  pmedia=atahd pdrv=Linux-2:/devtest/ psubdir=devtest pfix=fsck
In both cases got:
*** Puppy install partition not specified

Code: Select all

# blkid
/dev/sda3: LABEL="Linux-2" UUID="95a633e7-6fab-431d-9fa8-550812ab03f8" TYPE="ext4" PARTUUID="ee50f428-03"
The frugal install folder is /devtest and it's on /dev/sda3.....

What am I missing???
Last edited by peebee on Fri 09 Jun 2017, 07:12, edited 1 time in total.
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#17 Post by gyro »

@peebee,
try

Code: Select all

kernel /devtest/vmlinuz  pmedia=atahd pdrv=sda3:/devtest/ pfix=fsck
or

Code: Select all

kernel /devtest/vmlinuz  pmedia=atahd pdrv=Linux-2 psubdir=/devtest pfix=fsck
My current kernel line is

Code: Select all

  kernel /puppy/slacko/vmlinuz libata.noacpi=1 pmedia=atahd pdrv=Work psubdir=/puppy/slacko pfix=fsckp,nosave,copy
Unfortunately I commented out the line of code that prepends a "/" to psubdir so if specifying a "psubdir=" parameter it must start with a "/". (I'll fix that in the next version.)
Also best tot specify either a directory in the "pdrv=" parameter or with a "psubdir=" paramater, not both.
But I don't understand why "pdrv=sda3:/devtest/" didn't work.
I'll do some testing with boot paramaters like yours, and see if I can get it to fail.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

#18 Post by gyro »

The missing "/" on psubdir does cause havoc, only for me the kernel line

Code: Select all

  kernel /puppy/slacko/vmlinuz libata.noacpi=1 pmedia=atahd pdrv=sdb2:/puppy/slacko/ psubdir=puppy/slacko pfix=fsck
Still boots, but it loads only the puppy...sfs file, and hangs on shutdown.
But

Code: Select all

  kernel /puppy/slacko/vmlinuz libata.noacpi=1 pmedia=atahd pdrv=sdb2 psubdir=/puppy/slacko pfix=fsck
worked fine.
I have uploaded a new version that does prepend a "/" to psubdir if it's not specified.
It also definitely does propagate a directory specified as "pdrv=sdb6:/subdir/" as though "psubdir=/subdir" had been specified, provided that "psubdir=" is not specified.
So, these are the equivalent

Code: Select all

pdrv=Work:/subdir/
and

Code: Select all

pdrv=Work psubdir=/subdir
The new version does other stuff as well but I'll cover that in a later post.

gyro

gyro
Posts: 1798
Joined: Tue 28 Oct 2008, 21:35
Location: Brisbane, Australia

Announcing a new version

#19 Post by gyro »

Thanks to the folk who have tired to run this thing.
Trying to reproduce the problems that have been reported, has led to the fixing of some bugs.
Frustratingly, I still haven't been able to reproduce the same experiences.

But this post is to announce that a new version has been uploaded. (See first post for download utl's.)
Given BarryK's success with an overlay with a simple directory on disk as an rw layer,
I've decided to change the script in two ways.
1. It's now capable of doing the equivalent of a pupmode=5 boot using a directory on a disk as the rw layer instead of a directory in a tmpfs.
If you specify "pfix=nocopy", the sfs's are mounted directly off the disk and a save-directory is created and used as the rw layer.
As a consequence of this, on reboot the save-directory is found and it does a pupmode=12 boot.
This provides persistency without resort to "shutdowconfig". (Of couorse only on a Linux partition.)
In my testing I have a "shutdownconfig" that does nothing if "pfix=nosave" is specified, so I don't have to click on a "No" button on first shutdown.
2. The algorithm has been slightly modified, so that when processing each sfs one by one, each successfully mounted sfs is added to a list, not added to the stack.
Then there is a block of code that builds the stack using these lists of directories.
The idea is that when replacing aufs with overlays, it's all happening in a single block of code.

gyro

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#20 Post by peebee »

[FIXED] see below

Hi gyro - latest initrd.gz.....

I get:
*** Puppy install partition not specified

whatever combination of pdrv and psubdir I try.....

Is it because I'm using uuid??

Last attempt was:

Code: Select all

title Devtest (sda3/devtest)
  uuid 95a633e7-6fab-431d-9fa8-550812ab03f8
  kernel /devtest/vmlinuz  pmedia=atahd pdrv=sda3:/devtest/ pfix=fsck
  initrd /devtest/initrd.gz
Last edited by peebee on Fri 09 Jun 2017, 07:11, edited 1 time in total.
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

Post Reply