A SAVE-session to directory option added for PUPs [REOPENED]

[mavrothal]

Mav, you won't like my patch. It's about 1000 lines long and replaces /sbin/init entirely - it's called "Fatdog's /sbin/init"

I wish that was all needed
Add some rc.d, a dozen of fatdog_this scripts few more things that I miss, and then you are ready

Actually no ... its not really a case of patches but sorely in need of a full rewrite... look at what it needs to achive and write code to achieve that tidilly.

Let's see...
Would be nice to boot from CD, DVD, HDD, USB, MMC, network drive without any boot arguments. Find "magically" (no boot arguments) puppy data anywhere in the system in any form (savefile, savefolder, save partition, full install, underdog) and on any file system. Give the option to save anyway you want on anyfile system and media. Do all this fast and intuitively. Is this too much to ask

60,000 lines of code (bash/ash script I presume) suggests a very messy system that has far too many patches and cludges already... should not woof ce be an opportunity to look at things afresh... cleaner code, better features, removal of obsolete or unuseful ones.
Check out the slax boot/system wrappers...they have hardly changed at all over 9 years ...sort of got it right in the first place...i digress..back to puppy.

Wow. That is a fast "evaluation" of the code. (BTW the linux kernel changes 3500 likes a day).
Regarding a fresh start, please by all means go ahead. Put your code where your mouth is

[mikeb]

Well the init quadrupled in size and that mainly seems to be when installing to a folder was added. Puppy 2's init was much more streamlined and only really misses out folders which would be added with a variable here or there. Yes indeed rc.shutdown and others need some attention too.

But actually you are far too aggresive/hostile to deal with so I and others will leave you with you pleasant task of patching up the status quo.

I myself prefer an easier life....

Mike

[jamesbond]

But actually you are far too aggresive/hostile to deal with so I and others will leave you with you pleasant task of patching up the status quo.

I fail to see where the hostility is? In fact, Mavrothal is suggesting that nothing in Woof-CE is sacred, and he welcomes your changes (even if that means the entire Woof-CE has to be changed)

[mavrothal]

But actually you are far too aggresive/hostile to deal with so I and others will leave you with you pleasant task of patching up the status quo.

It is probably a matter of culture and/or language but I was literal.
You are obviously capable and knowledgable. So why not do it?
Actually just few days back I was suggesting is time for woof3.

[greengeek]

Save folder is implemented in Fatdog, since 620beta1 ...... For this to work, the filesystem of the partition must be a compatible Linux filesystem.

What is the reason for this limitation? If puppy can read say, FAT32, easily, what is the reason it can not store system files there? Is it the concern that such files could be modified/damaged by other operating systems using that partition?

Sometimes it is handy to be able to be able to keep the 2fs savefile on a FAT or NTFS partition - but why exactly the need for the linux filesystem restriction?

Let's see...
Would be nice to boot from CD, DVD, HDD, USB, MMC, network drive without any boot arguments. Find "magically" (no boot arguments) puppy data anywhere in the system in any form (savefile, savefolder, save partition, full install, underdog) and on any file system. Give the option to save anyway you want on anyfile system and media. Do all this fast and intuitively.

You forgot to mention it would be great for the persistence files to consume no more than 12Kb

[rufwoof]

@rufwoof - because it happens too frequent that people setup a small savefile and before you know it, it gets full. Save-to-directory "saves" you from this problem but it doesn't consume a whole partition like save-to-partition. And you can still have a full backup easily by copying the entire save directory, if you wish. But save-to-directory only makes sense for others who are already running Linux since it requires a Linux-compatible partition (not only ext2/3/4, you can have it with f2fs, xfs, jfs, reiserfs, btrfs, zfs, nfs, whatever-fs you can think of which is an Unix-compatible partition --- ntfs, vfat and cifs are specifically *excluded*).

Thanks JB.

I started using Puppy back in early March in anticipation of the April XP deadline. Initially I thought I'd just create a small Puppy LiveCD for online banking purposes - a sort of browser linux, and use XP for all other stuff. I haven't however even booted XP for over a month now. I'm finding that Puppy caters for all of my needs.

Initially I did try a savefile and experienced what you said - mozilla cache was filling it up. Played around with moving .mozilla outside of savefile, creating backups etc, but by then I'd more or less remastered a liveCD that had everything I needed and as I liked, so decided to just ram boot and sym link any changes I did make to outside of the savefile space. That's worked well for me and I've stayed with that since.

I don't want to full install as I'm more comfortable booting with the same pristine fresh image all of the time (I realise that could be achieved with frugal or full installs - but I'm ok with using a CD - especially as once booted the CD can be removed - leaving everything just running in RAM (no HDD's, on CD)).

I've a small Slacko based liveCD, 80MB (uses extreme compression so squeezes more into less space i.e. -b 1024K compression dictionary size) that boots quite quickly. The rest I've set up as a few clickable icons that each load multiple sfs's/pet's (word processing, audio/video editing etc). I just drag the EXTRA folder (the name I've given to the folder containing all of the sfs's) to / and then load them from there (in memory).

I store all images, music, docs etc on the HDD's, together with a script that sym links in the various selective persistent saves that I want - primarily program configuration (under /root) files.

Even though this ancient single core runs at 100% CPU when heavily loaded, Puppy manages the timeslicing/loading well and actual interaction is very snappy/quick. So snappy in fact that I've been unwilling to boot sluggish XP.

I've tried loads of other distro's, none work as well as Puppy as a LiveCD IMO. The developers/team are to be congratulated.

A big thanks to all concerned.

[jamesbond]

Save folder is implemented in Fatdog, since 620beta1 ...... For this to work, the filesystem of the partition must be a compatible Linux filesystem.

What is the reason for this limitation? If puppy can read say, FAT32, easily, what is the reason it can not store system files there? Is it the concern that such files could be modified/damaged by other operating systems using that partition?

Sometimes it is handy to be able to be able to keep the 2fs savefile on a FAT or NTFS partition - but why exactly the need for the linux filesystem restriction?

There are deep technical and usability problems around it, but I'll save you all that and be direct: you'll get kernel panics otherwise If you need to keep the foreign filesystem, stick to savefile.

[saintless]

Sometimes it is handy to be able to be able to keep the 2fs savefile on a FAT or NTFS partition - but why exactly the need for the linux filesystem restriction?

Not possible for full persistent because permissions and symlinking will be broken on NTFS and Fat.
But already included as option in Porteus for saving some type of files only with the strong advice to use ext save file instead NTFS or Fat folder:
http://www.porteus.org/component/conten ... lders.html

Toni

[gyro]

Assuming a frugal install using a "psubdir", why not have a save folder with the same filename as a save file but with a ".dir" extension instead of ".2fs", ".3fs" or ".4fs". And store it in the "psubdir" along with the system ".sfs" file.
In "init", if the pupsavefile has ".dir" extension then do a "mount --bind" insted of a "losetup" and a "mount".
At first shutdown, if "/mnt/dev_save" is a linux partition, create the save folder, and don't ask any questions.

gyro

[greengeek]

But already included as option in Porteus for saving some type of files only with the strong advice to use ext save file instead NTFS or Fat folder:
http://www.porteus.org/component/conten ... lders.html

Those Magic Folders sound interesting in that they can use separate savefiles for administrator and for guest (user) - I would really love to be able to save two separate "personalities" without having to save the massive overhead of two different 512MB savefiles:

One "personality" could have a NewZealand timezone and language setting, and the other would have US timezone and US language (to cater for the two usergroups in my house). No need for totally separate savefiles or usernames - we are happy to share/combine our data, but just a preference for different "personality" overlays that suit a particular user.

I don't know if that perspective would be better served by a save folder or compressed save file but just thought I'd chuck it in the ring...

EDIT :Sort of like this but maybe with an icon on the desktop that could "pull in" a save folder and overlay it. (maybe two icons on desktop - one a NZ flag and one a US flag)
So - not really like a full user environment - which puppy is not designed for - but just a 'partial overlay' that creates a different environment based on a save folder. (as usual just spitballin' here...)

[mikeb]

@ greengeek
did you not try the folder save (and sfs) and the multiuser option from the pup you had from me? ..a place I put my mouth
Then you can try out all this theory in practice.

As for multiple saves... well the choice of save file name boogie on any pup...but changing the read write layer is not really a feasible option. On the other hand some was playing with wiping and reloading the contents of tmpfs from an sfs ...crude but seemed to work.

Using a save folder requires a posix filesystem ... symlinks and permissions are 2 reasons that come to mind.

@giro
your are being too logical... you must leave

mike

[rufwoof]

Using a save folder requires a posix filesystem ... symlinks and permissions are 2 reasons that come to mind.

That's subjective to how much you want to have saved/persist-across-reboots. In my case I just save program config data i.e. /root/.notecase, /root/.openshot ...etc. Most if not all of those config files are relatively simple files - neither sym-linked nor permissions dependent.

Even for some system configuration changes sym's/permissions aren't critical, /root/.jwmrc-tray for instance.

When you extend up to wanting to save more changes, libs, /etc/rc etc then permissions/sym's become more important. But at that level shouldn't you really be thinking of doing a full install anyway.

If permissions/syms can't be maintained on the likes of ntfs and instead a container (sfs) has to be used, then how about a dynamic sfs, a pointer record included with the fixed part, that points to the end of (sfs) file and a size record immediately after EOF that identifies how much additional space beyond the 'end of file' that has been 'attached' (reserved) for read/write purposes. 1MB sfs file might have 100K additional filespace (1.1MB total filesize) where that last 100K is read/write. That additional read/write space could easily be dyamic (need to expand from 1.1MB to 1.2MB then just add the extra 100K onto the end and update a pointer/record (new sfs+additional filesize 1.2MB).

[mikeb]

are we talking about 2/3/4fs image files or squashfs archives?..not following that last bit. A sfs save simply grows according to the amount of data in the r/w layer if thats what you mean.

Subject in hand was using a save folder on a posix partition as a save layer.

Ok add a library by installing a pet .... if the r/w layer is not posix it breaks...end of story. Its linux...its built to run from a posix filesystem. The layered filesystem has to reproduce a posix filesystem for it to work...that requires posix layers.
Just like the initrd behaves as a posix filesystem.

Access to fat32 or ntfs is a convenience feature.... I can access ext2/3 partitions from windows with a convenience driver... windows itself still wants to exist on ntfs (not sure if fat32 ok now for vista/7/8)

File naming restrictions vary too I believe.

I wonder whats for dinner...

mike

[gcmartin]

There is great implementations floating around as this discussion progresses. I think members here get the picture that as we look at this Linux distro, there exist some measures that would allow a simple method of system management in addition to or in leu of a compressed save file or 2fs/3fs/4fs ...

The practice would work no matter if it was done on HDD or USB or DVD or SD or Blu-ray or ...

I have alway saved Linux stuff to linux compatibles. For all other stuff, mainly data and other content, the other filesystem options can and are used. But, again, NOT for linux needs where its symbolic linkages will preserve.

In offering a save folder solution for expansions primarily due to Package Management processes or required local changes in the RAM filesystem, when the options is given to the user, the shutdown subsystem will already know (thru its testings) whether the filesystem options are posix or not and ONLY allow a selection to the user to elect where on the posix to place the save-session information. This selection process, of course, ONLY happens once in the life of any reboot of the PUPs bearing the technology. The save-session would still continue to work the same (at a high level) as it does today; namely "at first shutdown, where so save ..."

It seems that this review is a good one and it seems that plausible solutions are being presented.

I want to weigh-in on the size issue: SIZE does NOT measure performance! We, users, see performance and as RAM pricing, new, use, or given away goes, its reasonable plentiful, today. The save-session processing that we are discussion is a one-time process that occurs on first-shutdown where a user is given options. We are NOT discussing FULL-INSTALLs, as there is no need for such in those systems. If someone/anyone changes a full-install to operate with a save-session, it no longer is a full-install.

For frugal and Live needs where PUP operates out of RAM, this thread has a place and is pertinence.

Hope this is helpful

[mavrothal]

Assuming a frugal install using a "psubdir", why not have a save folder with the same filename as a save file but with a ".dir" extension instead of ".2fs", ".3fs" or ".4fs". And store it in the "psubdir" along with the system ".sfs" file.
In "init", if the pupsavefile has ".dir" extension then do a "mount --bind" insted of a "losetup" and a "mount".
At first shutdown, if "/mnt/dev_save" is a linux partition, create the save folder, and don't ask any questions.

gyro

I wish was that simple.
pupsave is hardcoded to be a file not a directory.
If you eliminate this, then search through folders and subfolders for puppy files fails. So the logic must be revised.
This can be easily done if you sacrifice other install options but if you want to keep all the preexisting and add savefolders it becomes less "simple" (at least for me. I'm sure that sooner or latter someone will have code to offer)

[gcmartin]

Hello @Gyro

2fs/3fs/4fs are files that get created, then written into. The shutdown processing subsystem uses Linux utilities to create a file that looks like a posix partition, then writes the system's changes into that posix look-a-like. Once this is done, that file, namely 2fs/3fs/4fs can be place "undisturbed" on ANY known partiton (ext/NTFS/vFAT/FAT32/...etc) which a booting PUP can capably search on boot-up.

What is discussed here is an addition to shutdown-restart processing (persistence) change needs, where the changes (save-session) is a "real" folder for linux use, not one of these special files. The advantage and has been mention earlier, is that the primary system needs are NOT restricted to the size of the 2fs/3fs/4fs sizes....ever!

Hope this helps

[greengeek]

@ greengeek
did you not try the folder save (and sfs) and the multiuser option from the pup you had from me? ..a place I put my mouth
Then you can try out all this theory in practice.

No, i didnt get that far. I booted it live and had a look around but really it was just a cursory look with the intention of coming back to it (a bad habit of mine unfortunately ). I shall go back and do an install. Thanks for the reminder

[mikeb]

No, i didnt get that far. I booted it live and had a look around but really it was just a cursory look with the intention of coming back to it (a bad habit of mine unfortunately Embarassed ). I shall go back and do an install. Thanks for the reminder

ok well the save folder and archive is simply added to the shutdown options.
For multiuser rename /etc/slim.conf.bak to slim.conf and that triggers xwin to give a login screen for root or user. Use the create user wizard from the menu. Suggest test one thing at a time

mike

[rufwoof]

The advantage and has been mention earlier, is that the primary system needs are NOT restricted to the size of the 2fs/3fs/4fs sizes....ever!

This resize2fs manual page suggests that resizing might be possible whilst still being mounted (if ext3/4)

The resize2fs program will resize ext2, ext3, or ext4 file systems. It can be used to enlarge or shrink an unmounted file system located on device. If the filesystem is mounted, it can be used to expand the size of the mounted filesystem, assuming the kernel supports on-line resizing. (As of this writing, the Linux 2.6 kernel supports on-line resize for filesystems mounted using ext3 and ext4.).

Might then it be viable to monitor/dynamically (live) expand the savefile by 25% once it was >75% filled .... to equal effect (assuming ext3/4).

[gcmartin]

... Might then it be viable to monitor/dynamically (live) expand the savefile by 25% once it was >75% filled .... to equal effect (assuming ext3/4).

That intelligence would need to added to the running system somewhere; say, when attempting to add subsystem functionality via PPM.

But, if user had selected the option to use a folder, this intelligence would not be necessary. Every user, Apple/Microsoft/unix/Linux understands "drive level space management" solutions. ... everyone. Not so, when understanding of a "file" which is a filesystem which has run out of space and any problems which surface related to such. This is why the discussion is being pondered. In reality, a folder approach would reduce the number of questions that surface on the forum about issues related to 2fs/3fs/4fs.

Folders offer a clearer and clean solution. And certainly make for a universally understandable item to address. As is mentioned earlier about FATDOG, that approach attempts to address much of this. As does the other case where "under certain conditions" one is given the option of partition selection. But, this thread ponders the option of folder selection at shutdown to easy understandable save-session/persistence management for the RAM approach to Puppy operation. Primary, again, for both Live and Frugal experienced or non-experienced users.

Hope this is helpful.