Puppy 4.2

Promote Puppy !
Message
Author
PaulBx1
Posts: 2312
Joined: Sat 17 Jun 2006, 03:11
Location: Wyoming, USA

#101 Post by PaulBx1 »

Ah, good points. I especially like backing up to media other than where the pupsave is; I had just thought of a subdirectory on the same device.

Could this be done with a destination of a flash drive? It's my impression that flash drives still do not retain their name over boots though. Or not reliably, anyway.

Even better would just be an sdio or equivalent card that stays plugged into the machine all the time - convenient for laptops...

Actually I was not thinking of rebooting at all, but just doing it in the evening when you normally shut down anyway. Of course if you leave your machine on all the time, then a reboot is required. But still you have to go to bed sometime; so a reboot at that time would be no inconvenience at all. Oh, unless you have an encrypted pupsave!

This means the Menu>Shutdown menu would also need a "Reboot w/ Pupsave backup" entry.

User avatar
vtpup
Posts: 1420
Joined: Thu 16 Oct 2008, 01:42
Location: Republic of Vermont
Contact:

#102 Post by vtpup »

Paul,

Rebooting is often necessary when creating new apps, or installing (and removing) dependencies to try to get a piece of software to work, during testing, etc. It can drive you nuts with something complicated you're trying to get work.

And it is just at that time --while experimenting or developing when you might mess up your system. It would be great to be able to back up again just one session earlier when something goes wrong, rather than start all over again with a pupsave file you manually backed up maybe at the beginning of the day.

A delta would mean the shut down and reboot process would be relatively quick, as compared with writing a new full pupsave every time. In fact, this is why you generally don't back up enough, in this kind of process. Takes too much time.

You know, I don't think that backups actually need to be automatic on shutdown.... If the desktop Shutdown Menu had the option to save with backup, or not, you could just choose either without being asked which you want (and having to wait for the question, or even answer a question at all). This would help when you had to bail out quickly (like when you're late leaving for an appointment, or the lights start browning out, or the baby starts climbing the stairs, or......).

I'd normally just pick out "Shut down with backup" , or "Reboot with backup".

Maybe establishing the backup location shouldn't also be a wizard question, but a system setting. The point of this would be to avoid a closing dialog and a need to sit by the computer and answer something after the bye-bye button was pushed.

If the location setting was null on the liveCD or first install then a question could be asked about where it should go. Otherwise, it goes where you specified. Until and unless you change that setting.

PaulBx1
Posts: 2312
Joined: Sat 17 Jun 2006, 03:11
Location: Wyoming, USA

#103 Post by PaulBx1 »

If the desktop Shutdown Menu had the option to save with backup, or not, you could just choose either without being asked which you want
That's what I meant. That having those two shutdown options would eliminate the need for any dialog. Probably didn't make that clear in that post above, as I was thinking while I was typing.
Maybe establishing the backup location shouldn't also be a wizard question, but a system setting. The point of this would be to avoid a closing dialog and a need to sit by the computer and answer something after the bye-bye button was pushed.
Well, the question about backup location I was envisioning asking only once, in the first dialog when the pupsave is initially created. And it would only affect a variable assignment in PUPSTATE, which could be changed at any time later by the user.

I was not envisioning asking the location question on every shutdown. That would get annoying.

So bottom line, the shutdown would look just like it does now, except you have a couple extra options in the shutdown menu.

The only other thing, I was wondering how to generate more than one backup copy. In other words, having a ";-1" version, a ";-2" version, etc (if you are familiar with Vax VMS notation, heh). That would be an extra though, much less important than having a backup in the first place. Probably could be done at boot in rc.local using "nice" and keeping it in the background so you wouldn't notice the copy going on. I suppose you could use rsync there too to keep the thrashing down. But I don't want to get too greedy. If I have a backup option in the shutdown menu I will be very happy.

User avatar
vtpup
Posts: 1420
Joined: Thu 16 Oct 2008, 01:42
Location: Republic of Vermont
Contact:

#104 Post by vtpup »

Well Paul, I was thinking that each "backup" was separate. Since it was only a diff, it would be small, relatively. To go back any distance in time you apply as many diffs back as you need to go in time. Basically like an undo list in a graphics program. You never overwrite any of them. These are located on hard media and are individually write-once. When the disk is full it gets a volume label and a new disk is asked for.

Here is also some interesting info on another somewhat similar backup method:

http://www.mikerubel.org/computers/rsyn ... index.html

The main problem I see in his is that I believe he's mainly talking about backing up a Home directory, not an entire operating systyem. But I haven't read it carefully enough to be able to say for sure. The basic problem I'm guessing is how to save something while you are actively using it. I don't know if this is solved in this method or not. I do understand how during a shut-down, a low level process could save the OS without iself altering it while acting. But I wonder if this is possible at a higher level?

PaulBx1
Posts: 2312
Joined: Sat 17 Jun 2006, 03:11
Location: Wyoming, USA

#105 Post by PaulBx1 »

From that article:

Code: Select all

rm -rf backup.3
mv backup.2 backup.3
mv backup.1 backup.2
cp -al backup.0 backup.1
rsync -a --delete source_directory/  backup.0/
That's more or less what I was thinking of. I don't quite understand the details (that the latest copy is the "big" one and the others are just differences). I haven't used rsync before. Thanks for that link, by the way.
The basic problem I'm guessing is how to save something while you are actively using it... But I wonder if this is possible at a higher level?
Well, I hadn't thought this through either. I did poke around in the rc.shutdown script but nothing jumped out at me. I was thinking of doing it not when the pupsave was being used, but after it had been umounted and disassociated from the loop device (if possible). Then just copy or rsync the file. Maybe copy the backup script into ramdisk first, so it is not inconvenienced by being inside the pupsave! :) I'm pretty hazy about this as I don't know the shutdown process very well.

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#106 Post by Pizzasgood »

Woah, cool! So that's what hard links are good for.

That technique wouldn't work for backing up a save file unless you pulled the contents out of it, since it's just one big file. The problem then is that it would need a linux partition to store them on, so people who run it on Windows-only machines would be out of luck. A way around that could be to set up another filesystem image for just backups: pup_back.2fs. Since you're not running from that file, it could be resized without having to reboot first like with the pup_save.2fs file. It would just need to be unmounted if it isn't already. So it could be kept close to the size of the actual data inside to avoid wasting harddrive space.

Using the .2fs extension would still let you just click it to mount. Though maybe it would be a good idea to add a new extension .ROfs, and make the run-action mount .ROfs files as read-only. That way if somebody clicks it to mount it, they won't modify anything.


That method might be overkill though. Fewer opportunities for errors with just copying the pup_save.2fs file itself, though it's more wasteful space-wise. Maybe use something like diff or xdelta to only store the differences (the distinction here is that I'm referring to the differences between just two big files, as opposed to an entire copy of every small file that has changed).
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

Puppy 4..2alpha "Deepthought" ISO uploaded...

#107 Post by WhoDo »

...see HERE for mirrors and details.

Enjoy! :P
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

PaulBx1
Posts: 2312
Joined: Sat 17 Jun 2006, 03:11
Location: Wyoming, USA

#108 Post by PaulBx1 »

That technique wouldn't work for backing up a save file unless you pulled the contents out of it, since it's just one big file.
Um, it's my impression that rsync works with pieces of files, rather than whole files. From the rsync site: "It does this by sending just the differences in the files across the link..." So it should work just fine with pupsaves.

I do wonder a bit about encrypted pupsaves. If you change one character in the cleartext state, IIRC you change a block (of some unknown-to-me size) within the encrypted file, but not the entire file. So I think it should work with encrypted pupsaves too, but not as quickly as with unencrypted ones.

I re-read that link and now think I have a handle on what is going on here. Still seems like magic though! :)

In rc.shutdown, wouldn't this be done just before the "busybox umount -ar > /dev/null 2>&1" at the end? Something like this:

Code: Select all

umount /initrd/pup_rw
sync
rm -rf $PUPSAVEBACKUPDIR/pupbackup.3
mv $PUPSAVEBACKUPDIR/pupbackup.2  $PUPSAVEBACKUPDIR/pupbackup.3
mv $PUPSAVEBACKUPDIR/pupbackup.1  $PUPSAVEBACKUPDIR/pupbackup.2
cp -al $PUPSAVEBACKUPDIR/pupbackup.0  $PUPSAVEBACKUPDIR/pupbackup.1
rsync -a --delete --checksum $PUPSAVE/   $PUPSAVEBACKUPDIR/pupbackup.0/
busybox umount -ar > /dev/null 2>&1
Well, I guess that wouldn't work if 2 different pupsaves were being used on different boots? At least you want to keep the corresponding backup directories distinct.

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#109 Post by Pizzasgood »

I didn't mean there'd be a problem with rsync. I mean using hardlinks wouldn't work, because that technique only prevents you from having multiple copies of identical files - instead you have multiple hardlinks to the same file. When a file changes, the next backup will include a real copy of that file rather than the hardlink. So if you only have one file, every time it changes it will be a complete copy, and you don't save any space.

Looking at the code you posted, it looks like you're backing up the whole subdirectory that Puppy was (theoretically) installed into. In that case, using the hardlinks would save space by preventing multiple copies of vmlinuz, initrd.gz, and pup_xxx.sfs.

Also, if you do use the hardlink method, I don't believe that rsync would give you a speed increase for the part when it copies in a new file over an old one. Normally it would, due to the differences deal, but if the destination file is a hardlink, I believe it would have to copy all the data. Reasoning: The hardlinked file's data only exists in one single place. The two names are just different names to the same file. For it to make the two hardlinks into separate files, the data needs to exist in two separate places. It can't simply drop in the changed data and leave the unchanged parts of the file as links to the original, since links must be done on entire files. So it has to either copy in the entire file rather than the differences, or it must copy the unchanged parts from the old file. Either way it has to do the same amount of writing to the disk as if it had copied the whole thing.
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
vtpup
Posts: 1420
Joined: Thu 16 Oct 2008, 01:42
Location: Republic of Vermont
Contact:

#110 Post by vtpup »

Yes, for a complete copy of the personal save file that method in the article wouldn't make sense. I wasn't thinking properly about that.

But there might be something useful in it for the case where the frugal was mounted and you were periodically backing up to another mounted version on say an hourly basis, while doing development work. Then you would only lose an hour's worth if you crashed and burned, and the saves would be to individual files and would be rapid and small (hopefully) since only the diffs would be infolved.

Might also be nice to be able to do it with a key combination on demand as well. Kinda like <ctrl> <s> does in some programs, except in this case it backs up the whole OS state!

It would then relieve you of the need to periodically back out of what you are doing and close, reboot another puppy or pfix=ram, save the personal file, close down again and reboot to the original development state. I doubt anybody does that on an hourly basis manually while developing something. Just too painful.

The incremental backups certainly would make sense for a full Puppy HD install. In fact that would make it a LOT tougher system.

I realize though that while running, the HD is in flux, so maybe it's just not possible to do a backup then. Unless you could temporarily halt file changes.

Failing that, and to return to Paul's original request, what would be really helpful to frugalites would be a choice to shutdown (or reboot) with backup of the personal save file included in the shutdown process.

This could be a choice on the main Menu shutdown menu.

PaulBx1
Posts: 2312
Joined: Sat 17 Jun 2006, 03:11
Location: Wyoming, USA

#111 Post by PaulBx1 »

Well, it looks like I didn't understand it after all. Now it makes sense, that you say it refers to multiple files and not just one.

Anyway, this multiple backup was a side issue. The main thing is having at least ONE backup, generated from the menu selection at shutdown as you say. If I want more copies I can just do it myself in rc.local I guess.

User avatar
Max Headroom
Posts: 421
Joined: Wed 28 Jun 2006, 07:17
Location: GodZone Kiwi
Contact:

G'day Technosaurus How's Y'r Project Going?

#112 Post by Max Headroom »

technosaurus wrote:@vtpup - using Puppy's layered file system it is easy to do both by having barebones + a buildback sfs ...using my example before, just have a symlink to standard.sfs in the /pupXXX directory

I should have a demo build around the new year thanks to some help from pizzasgood with a bash script on this thread
http://www.murga-linux.com/puppy/viewtopic.php?t=36886

The boot menus will be derived from Fancy and Retro Pup - highly modified to include only one vmlinuz, init and "zdrv"(actually an sfs of all files shared by all of the puplets - to save space) I plan to try and include as many smaller puplets as possible + extra sfs modules to top off the iso (basically I am targeting smaller puplets that did a lot of work getting additional window managers & UI all set up hopefully with a good range of browsers and core apps - the rest can be pets & sfs modules until if/when I decide to do a DVD)
Looking Forward 2 This Mate, in Fact I was intending 2 Do Sumthing Simila' if Not as Sophisticated. Bring it on!

Post Reply