A SAVE-session to directory option added for PUPs [REOPENED]

A home for all kinds of Puppy related projects
Post Reply
Message
Author
gcmartin

A SAVE-session to directory option added for PUPs [REOPENED]

#1 Post by gcmartin »

After the thread's efforts were accepted, GCMartin wrote:This addition to PUPPY allows session changes to be saved to a directory (folder). Its primary advantage over the older "save to a file" method is that the size restriction of saving to a file disappears.

Thanks to everyone who have participated in bringing this option to PUPPY Linux from this time forward.

This thread is requested closed
Over the years our systems have grown in every dimension: RAM size, disk size, disc size, USB size, SD technology, etc. not to mention that other factors like LAN and WAN speed increases have occurred in the home and workplace. Puppy got it start in an age when existing resources at the time were extremely constrained.

The current save-session strategy is based upon a very good strategy on one hand, but, I feel it can be improved. Here's an advantage I see:
Instead of the save-session being "stuffed" into a "disk mimicking" container file as it is done for frugal/Live running systems, could it be created, unstuffed, in a single partition instead of the "constained container" of today? It is constrained because it is "LOCKED" at a selected size. Removal of this locked constraint in today's system would allow users greater flexibility in use of their system(s).

If this is allowed, one MAJOR problem will be averted. Today, that major problem is expanding a save-session disk image, on the fly. If partition save-session technology is offered, many/most users will NEVER need to do any save-session expansion. Further, if a partition need expansion, gParted does this in non-destructive ways as well as the ability to copy/clone a partition is simple and understood by most.

My questions
  • Can this option to users be done today for a save-session?
  • Is there anything missing in the logic for such an option to be available for save-session use?
    Note: Further, allowing a save-session to a folder on permanent storage versus a dedicated partition, would also accomplish the same objective if I am correct.
Any guidance, understanding, or demonstration that may exist where a booting PUP distro would see and use a partition/folder instead of a file (as is done today *.4fs) to present a RAM based operation as it does today for frugal installations would be helpful. This intends NOT to be a full-install approach, rather it intends to continue Frugal/LiveDVD RAM system operations, as it occurrs today; but, without the mentioned constraint.

Please offer any idea(s) you feel are helpful.
Last edited by gcmartin on Mon 05 Oct 2015, 02:24, edited 3 times in total.

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#2 Post by 8-bit »

When one first gets the message to set up a pupsave file, they are given options as to where to save the file and even if they want to use a file with an option of using the entire partition.
I do not know if the partition selection is restricted to the partition Puppy is installed on or one can select any partition on any drive.
In effect, this comes very close to being like a full install.
But when booting, the personal information, additions, etc is loaded into memory from the selected partition rather than from a pupsave file and then the Puppy SFS is loaded and merged with it.

Is this similar to what you are requesting?

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#3 Post by rufwoof »

The savefile needs to be a ext format (symbolic links etc). Creating a sfs savefile enables that savefile to be stored on other file system types (NTFS etc).

If you have a ext partition, then one way (the way I do it) is to be selective as to what gets saved or not. I boot from a LiveCD in ram mode (no savefile, so 'savefile' space is all of RAM + swap (1.7GB in my case)), and have a script that creates symbolic links to copies of the directories/files I want to persist across reboots that are stored on the HDD ext partition (linked to the directories/files that are inside Puppy).

For example maybe have a copy of /root/.mozilla on HDD and rm /root/.mozilla and ln -s /mnt/home/.mozilla /root/.mozilla

I wrap a simple test around that to see if the symbolic link already exists or not (in case accidentally run twice) - something like (where curdir is set to /mnt/home)

Code: Select all

cd /root
echo l >./.mozilla/loadedtest
if [ -f $curdir/.mozilla/loadedtest ];
then
   # looks like already previously been loaded
   # just remove the test file
   rm $curdir/.mozilla/loadedtest
else
   cd /root
   rm -r -f .mozilla
   ln -s $curdir/.mozilla .mozilla
   cd $curdir
fi
i.e. it creates a file in /root/.mozilla and if that file appears in /mnt/home/.mozilla then the symbolic link is already in place, otherwise it creates the symbolic link.

So I boot from LiveCD in RAM only mode (puppy pfix=ram pmedia=cd, HDD's remain totally untouched and it all runs totally from RAM/CD i.e. online banking mode (fresh opsys and browser)). Or I run a script after having booted which loads some sfs's and pet's and creates all of the desired symbolic links (general usage desktop and (specific/targeted) changes persist across reboots).

Mikeb put me onto yet another way to do it - he copies /init/pup_pw (again selected parts (not tmp, dev, proc ..etc)) to a sfs and copies that back in again after the next reboot (changes persist).

Puppy's great for its flexibility and choices, and IMO the best choice of LiveCD with optional persistence (small size = speed and range of choices for persistence (none, some, all)).

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

Re: Request for SAVE-session option to remove a constraint

#4 Post by mavrothal »

gcmartin wrote: My questions.
The option to save to partition while booting with an SFS is there.
However, you can not do that when booting from a CD (Does not make sense).

To implement this do the following.
- Boot from a CD and Install in an (empty) partition.
- Run grub4dos
- Shutdown without saving
- Remove CD
- Boot from partition (will be a first run)
- On shutdown you are given the option to save to partition

See sceeny
Attachments
save_options.png
(28.78 KiB) Downloaded 4420 times
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#5 Post by mikeb »

Not make sense from a CD??? no more or less than a save file really...odd statement.

Actually its not quite as neat as it could be... ie the often cited use of a save folder rather than the root of a partition.... that then does allow coexistance with the puppy sfs AND other semi frugal installs.

You get the same freedom of space without hogging the partition.

My save folder ala slax is now 5 years old.... never had a problem generally and definitely never ran out of space.

The adjustment even to puppies init script is tiny compared to other funky options that could exist.... (hint ..use a bind mount)

Note for earlier...the partition like the save file is mounted and added to the union layer..nothing is loaded to ram.

The save folder option is what I use for my older desktops which have less ram to play with....I gave up on save files some years ago.

mike

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

#6 Post by mavrothal »

mikeb wrote:Not make sense from a CD??? no more or less than a save file really...odd statement.
The reason that does not make sense (to me) to have a partition (as suggested) as a "savefile" to removable media is that is a source of trouble when you use different puppies. A savefile on the other hand can indeed accommodate several different puppies on the same partition without a problem. And this is something that can be done with current puppies.
mikeb wrote:Actually its not quite as neat as it could be... ie the often cited use of a save folder rather than the root of a partition....
.....
The adjustment even to puppies init script is tiny compared to other funky options that could exist.... (hint ..use a bind mount)
Savefolder would make sense of course (as in the case of different savefiles).
Since you apparently know how to implement multiple savefolders in a partition please share your experience and provide some working code for puppy.

Edited for clarity
Last edited by mavrothal on Sat 03 May 2014, 10:50, edited 2 times in total.
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==

gcmartin

#7 Post by gcmartin »

The idea of folders for a given PUP is a better one. Instead of having the system compressing and decompressing SFSs, I would like to have session saves direct to folders (even encriypted in some case if necessary).

With all the HDD space, a SFS compressed file solution is probably unnecessary for many of us (when PUPPY got started years ago, HDD size-cost was an issue. Most PCs built since 2000+ have ample space for PUP's meager save-sessions needs) and the overhead, although, minor is some cases, is unnecessary given a choice to the user.

Thanks @Mavrothal as that picture you show is similar to what is asked.

After thought about what each of you have shared, a folder selection on a Linux partition would provide a better solution where folder could exist for each PUP one want to create.

Today, PUPs "key" on some prefix followed by 'save' (L64save.4fs for example). Could PUP's save session name and key of a folder with such naming instead of a SFS file? This seemingly would preserve the logic in save-session processing while extending the option for user selection.

Lastly, if the forum perfect this, what chance is there that it would land in WOOF-CE so that future PUPs would have a consistent offering to users at shutdown processing?

Here to help

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#8 Post by jamesbond »

Save folder is implemented in Fatdog, since 620beta1 (over a year ago ) http://www.murga-linux.com/puppy/viewtopic.php?t=84372. We did get the idea from slax. For this to work, the filesystem of the partition must be a compatible Linux filesystem.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

#9 Post by mavrothal »

gcmartin wrote: if the forum perfect this, what chance is there that it would land in WOOF-CE so that future PUPs would have a consistent offering to users at shutdown processing?
The chances are excellent, particularly if mikeb, jamesbond or anyone else provides any patches :wink:
You see, nobody "owns" woof-CE. If people contribute will grow, if not will fade away. Is a "democracy" now... :lol:

Edit: Added second line/paragraph.
Last edited by mavrothal on Sat 03 May 2014, 12:27, edited 3 times in total.

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#10 Post by mikeb »

Yeah it was slax that gave me the idea too.. I first tested it out on puppy 2.12 and then added it to others....so its not like this is cutting edge.

It makes double sense since most users now use a puppy folder anyway so in slax style it just needs a folder added in there really.
In my case I just made a folder called {$PUPPYVERSION}_save/
but since many pups now share the same version numbers (!) something more verbose makes sense if NOT using the pup folder.

Removable media and storage on permanent...well thats a whole can of worms really...I tend to see things as either / or ... either a live CD with perhaps persistence to itself OR install to a medium be it usb or hard drive....certainly would simplify the boot script.
If a specifically named folder is used then same rules apply as to a save file...ie multiple saves should not be a problem.... now as to why we all have multiple pups thats another story :D

As to script to mount a save folder... I am not on puppy but will post it when I am... basically added the detection of a version named folder and then bind mounted it to the union. Like with my other save options there is more code involved to add the option to the shutdown choices than to implement the actual save method. This one slipped into the code for a save file detection.

mike

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#11 Post by 8-bit »

mavrothal wrote:
gcmartin wrote: if the forum perfect this, what chance is there that it would land in WOOF-CE so that future PUPs would have a consistent offering to users at shutdown processing?
The chances are excellent, particularly if mikeb, jamesbond or anyone else provides any patches :wink:
You see, nobody "owns" woof-CE. If people contribute will grow, if not will fade away. Is a "democracy" now... :lol:

.
That is true to an extent.
For it to be implemented in Woof-CE, would it have to be approved by our benevolent dictator (maybe retired), Barry.

And just being curious, how much different is Woof-CE from Barry's official Woof?

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

#12 Post by mavrothal »

8-bit wrote:
mavrothal wrote: You see, nobody "owns" woof-CE. If people contribute will grow, if not will fade away. Is a "democracy" now... :lol:
That is true to an extent.
For it to be implemented in Woof-CE, would it have to be approved by our benevolent dictator (maybe retired), Barry.
Actually is absolutely true!
Barry, although very welcome, has nothing to do with woof-CE. Looks like he is indeed retiring.
"You are on your own kids" :lol:
8-bit wrote:And just being curious, how much different is Woof-CE from Barry's official Woof?
About 60.000 lines of code...
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#13 Post by rufwoof »

gcmartin wrote:The idea of folders for a given PUP is a better one. Instead of having the system compressing and decompressing SFSs, I would like to have session saves direct to folders (even encriypted in some case if necessary).

With all the HDD space, a SFS compressed file solution is probably unnecessary for many of us (when PUPPY got started years ago, HDD size-cost was an issue. Most PCs built since 2000+ have ample space for PUP's meager save-sessions needs) and the overhead, although, minor is some cases, is unnecessary given a choice to the user.

Thanks @Mavrothal as that picture you show is similar to what is asked.

After thought about what each of you have shared, a folder selection on a Linux partition would provide a better solution where folder could exist for each PUP one want to create.

Today, PUPs "key" on some prefix followed by 'save' (L64save.4fs for example). Could PUP's save session name and key of a folder with such naming instead of a SFS file? This seemingly would preserve the logic in save-session processing while extending the option for user selection.

Lastly, if the forum perfect this, what chance is there that it would land in WOOF-CE so that future PUPs would have a consistent offering to users at shutdown processing?
Why the need for change?

Currently the choice spans from saving nothing (ram boot with no save) to everything (full install), with any other combination between that you choose. Personally I prefer ram boot with no savefile, but do symbolic link a few files/directories for where I want persistence. Others are content with full installs. Frugal is somewhere in the middle (like having a difference file where only changed files/directories are recorded). And you can save to a wide range of medium - usb, cd, hdd.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#14 Post by jamesbond »

mavrothal wrote:
8-bit wrote:
mavrothal wrote: You see, nobody "owns" woof-CE. If people contribute will grow, if not will fade away. Is a "democracy" now... :lol:
That is true to an extent.
For it to be implemented in Woof-CE, would it have to be approved by our benevolent dictator (maybe retired), Barry.
Actually is absolutely true!
Barry, although very welcome, has nothing to do with woof-CE. Looks like he is indeed retiring.
"You are on your own kids" :lol:
In other words, Barry has nothing to do with Woof-CE. Woof-CE has its own gatekeepers, which I assure you, is more generous when accepting contributions (that being its lifeblood, you see) :D

@rufwoof - because it happens too frequent that people setup a small savefile and before you know it, it gets full. Save-to-directory "saves" you from this problem but it doesn't consume a whole partition like save-to-partition. And you can still have a full backup easily by copying the entire save directory, if you wish. But save-to-directory only makes sense for others who are already running Linux since it requires a Linux-compatible partition (not only ext2/3/4, you can have it with f2fs, xfs, jfs, reiserfs, btrfs, zfs, nfs, whatever-fs you can think of which is an Unix-compatible partition --- ntfs, vfat and cifs are specifically *excluded*).
The chances are excellent, particularly if mikeb, jamesbond or anyone else provides any patches
Mav, you won't like my patch. It's about 1000 lines long and replaces /sbin/init entirely - it's called "Fatdog's /sbin/init" :lol: And then we'll have a bigger problem :wink: Mikeb's one has better chance because it comes from puppy's /sbin/init, at least.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#15 Post by mikeb »

Mav, you won't like my patch. It's about 1000 lines long and replaces /sbin/init entirely - it's called "Fatdog's /sbin/init" And then we'll have a bigger problem Mikeb's one has better chance because it comes from puppy's /sbin/init, at least.
Actually no ... its not really a case of patches but sorely in need of a full rewrite... look at what it needs to achive and write code to achieve that tidilly.

60,000 lines of code (bash/ash script I presume) suggests a very messy system that has far too many patches and cludges already... should not woof ce be an opportunity to look at things afresh... cleaner code, better features, removal of obsolete or unuseful ones.
Check out the slax boot/system wrappers...they have hardly changed at all over 9 years ...sort of got it right in the first place...i digress..back to puppy.

I threw out the puppy 4+ init that most are using as after finding some unpleasant bugs I also realised that modifying it to add features was a simply an unfeasible nightmare. In my case I took the puppy 2 init...cleaned it out and took out the snapmerge stuff and then added my options. Modifying is pretty easy as the source it was built from had a pretty logical modular approach. So I can only say such and such works... sfs to ram save, save folder, multiple sfs and so on ... other derivatives have added similar... and as these mods work well then perhaps main stream puppy could benefit ... since woof is the ONLY way anything like that is allowed in...here we are.
I tested loading the full system to ram for Quirky ... the actual implentation will not be the same as I did the test using puppy 4.12 but it showed that it was feasible plus gave a line or three of code of the core of the technique. For quirky thats up to the writer to adapt the technique into his code.

So I have an init of 22k .... works for me...tested my experiments and of no use as it is for anyone else most likely being an individual of one I can only test on what I have around me. Although init is fairly portable they do normally have to be tweaked to match each kernel's release's quirks.

Ok the save folder is detected in the save detection code looking for the name pattern

Code: Select all

	if [ -d /mnt/data/${PUPPYVERSION}_save ]; then
		PUPSAVE="${1},${2},/${PUPPYVERSION}_save"  #use folder..still mode 12
		PUPMODE=12
	fi
and in the load save file section we have...

Code: Select all

	if [ -d $SMNTPT$SAVEFILE ]; then #using a folder so no loop needed
		echo -n "Mounting ${SAVEFILE} on ${1}..." >/dev/console
		mount -o bind $SMNTPT$SAVEFILE $1;check_status $?  
then the rest is the original save file mount script
note later busybox uses --bind and ${1} is normally /pup_rw.
Bind is used as the dev_save is already mounted plus we need to effectively mount our save folder.

mike

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

#16 Post by mavrothal »

jamesbond wrote:Mav, you won't like my patch. It's about 1000 lines long and replaces /sbin/init entirely - it's called "Fatdog's /sbin/init" :lol:
I wish that was all needed :shock:
Add some rc.d, a dozen of fatdog_this scripts few more things that I miss, and then you are ready :P
mikeb wrote:Actually no ... its not really a case of patches but sorely in need of a full rewrite... look at what it needs to achive and write code to achieve that tidilly.
Let's see...
Would be nice to boot from CD, DVD, HDD, USB, MMC, network drive without any boot arguments. Find "magically" (no boot arguments) puppy data anywhere in the system in any form (savefile, savefolder, save partition, full install, underdog) and on any file system. Give the option to save anyway you want on anyfile system and media. Do all this fast and intuitively. Is this too much to ask :lol:
mikeb wrote:60,000 lines of code (bash/ash script I presume) suggests a very messy system that has far too many patches and cludges already... should not woof ce be an opportunity to look at things afresh... cleaner code, better features, removal of obsolete or unuseful ones.
Check out the slax boot/system wrappers...they have hardly changed at all over 9 years ...sort of got it right in the first place...i digress..back to puppy.
Wow. That is a fast "evaluation" of the code. (BTW the linux kernel changes 3500 likes a day).
Regarding a fresh start, please by all means go ahead. Put your code where your mouth is :wink:
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#17 Post by mikeb »

Well the init quadrupled in size and that mainly seems to be when installing to a folder was added. Puppy 2's init was much more streamlined and only really misses out folders which would be added with a variable here or there. Yes indeed rc.shutdown and others need some attention too.

But actually you are far too aggresive/hostile to deal with so I and others will leave you with you pleasant task of patching up the status quo.

I myself prefer an easier life....

Mike

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#18 Post by jamesbond »

mikeb wrote:But actually you are far too aggresive/hostile to deal with so I and others will leave you with you pleasant task of patching up the status quo.
I fail to see where the hostility is? :? In fact, Mavrothal is suggesting that nothing in Woof-CE is sacred, and he welcomes your changes (even if that means the entire Woof-CE has to be changed) :oops:
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
mavrothal
Posts: 3096
Joined: Mon 24 Aug 2009, 18:23

#19 Post by mavrothal »

mikeb wrote: But actually you are far too aggresive/hostile to deal with so I and others will leave you with you pleasant task of patching up the status quo.
It is probably a matter of culture and/or language but I was literal.
You are obviously capable and knowledgable. So why not do it?
Actually just few days back I was suggesting is time for woof3.
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#20 Post by greengeek »

jamesbond wrote: Save folder is implemented in Fatdog, since 620beta1 ...... For this to work, the filesystem of the partition must be a compatible Linux filesystem.
What is the reason for this limitation? If puppy can read say, FAT32, easily, what is the reason it can not store system files there? Is it the concern that such files could be modified/damaged by other operating systems using that partition?

Sometimes it is handy to be able to be able to keep the 2fs savefile on a FAT or NTFS partition - but why exactly the need for the linux filesystem restriction?
mavrothal wrote:Let's see...
Would be nice to boot from CD, DVD, HDD, USB, MMC, network drive without any boot arguments. Find "magically" (no boot arguments) puppy data anywhere in the system in any form (savefile, savefolder, save partition, full install, underdog) and on any file system. Give the option to save anyway you want on anyfile system and media. Do all this fast and intuitively.
You forgot to mention it would be great for the persistence files to consume no more than 12Kb :-)

Post Reply