snap2 rotating snapshot backups for Puppy

Miscellaneous tools
Message
Author
User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#21 Post by lstandish »

Snap2 started as a program to do automatic non-interactive backup (via cron), and it is still oriented toward running a predefined backup job. However, you can get the result you want (fine-grained control over which partitions or sets of directories to back up) by creating additional backup 'sets'.

You could create a new backup set to backup any group of directories that you want. The result is that you back up only part of the partitions or folders. If you do this, you should however specify a different storage folder.

You can also set up a menu entry to execute a backup (via execution of snap2shell), and you can of course specify which backup set to back up, as an argument to snap2shell.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#22 Post by Sylvander »

1. "you can get the result you want (fine-grained control over which partitions or sets of directories to back up) by creating additional backup 'sets'"
(a) Are there any instructions on how to do that?

(b) Seems I'd need to create various backup sets for every possible combination of partitions.
Not a very elegant solution.
I don't see myself doing that. :(

(c) Or else configuring the program anew [to specify the partitions to backup] each time I run it.
Wouldn't want to do that either. :(

2. "The result is that you back up only part of the partitions"
I don't think I'd ever want to backup only PARTS of chosen partitions.
I'd want to backup only certain chosen partitions, but backup ALL of the contents of those.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#23 Post by lstandish »

Yes, you'd need to create a backup set for every combination of partitions you would back up. However, that's easy: A new set is created on the ADVANCED SETTINGS page by just typing in a new set name and clicking 'Change'. Then you follow the 3 step instructions at the top to configure the new backup set.

As I said, this is designed for those who do regular backups of predefined sets of directories (partitions in your case). It is really not intended for sporadic use or one-time backup. Wouldn't you want to back up your partitions (the same ones) on a regular basis?

In order to assist and encourage regular backup, snap2 will set up cron tasks to execute a backup set on certain days at a certain time.

I just finished the modifications to the code to make it not use the default exclusion patterns when there is no custom exclusion pattern and the 'Add default exclusions' is unticked.

Also, starting with the latest version (3.2-18), backup paths that have a custom exclusion pattern list are marked. That was a glaring deficiency, I think. 3.2-18 should be available a few minutes after I make this post.
Last edited by lstandish on Thu 21 Jan 2010, 17:47, edited 2 times in total.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#24 Post by lstandish »

duplicate post removed

User avatar
alienpup
Posts: 9
Joined: Sun 18 Oct 2009, 16:18
Location: Arizona, USA

#25 Post by alienpup »

lstandish wrote:Yes, you'd need to create a backup set for every combination of partitions you would back up.
I've run into a very significant issue when attempting to backup pup_save files using any rsync-based utility. The problem is that Puppy does not "touch" the pup_save while/after updating it. Because the file's time stamp does not change, rsync does not detect that the file has changed. The pup_save is backed one time and one time only.

I've solved the problem in a backup script of my own by explicitly touching the pup_save file before running rsync.

As this problem likely exists in all Puppies and could cause other problems, it probably ought to be brought to Barry's attention.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#26 Post by lstandish »

rsync decides to back up a file when it has changed in size or in last-modified time. If pupsave files don't get a fresh timestamp when Puppy updates them, I think that just about any backup utility (not just rsync) would have a problem detecting that it should be backed-up.

Your use of touch just before running a backup sounds like the best workaround until Barry fixes this.

By the way, since rsync transmits only the (compressed) differences in files when backing up, it is particularly suited to backing up big files such as pupsave files.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#27 Post by Sylvander »

1. "you'd need to create a backup set for every combination of partitions you would back up"
OK, beginning to get a feel for this. :D
(a) Within the "Advanced Settings" pane...
Made an "Active Backup Set" [new name] for [almost] each individual partition [sda1+sda5 (Windows+its-data-files), sda3 [BoxPup], sda6 [music], sda7 [photos]]...

(b) After giving each new name [e.g. sda1,sda5.set], clicked the "Change" button and then completed the "3 easy steps".
See the screenshot below of the storage folders I made/used.
One folder for each partition.
Might need to eliminate the default that makes a backup of ALL partitions to the "snap2" folder.
Would that result in a time-consuming repeat of first-time backing-up of all the individual partitions?

(c) So now I can choose to backup partitions individually, and when I like.
Might it be possible [upon running the program after configurations have been completed] to make that choice more prominent?
e.g. When the user runs the "Backup Now..." from the menu...
If there's more than 1 backup set configured...
To have a separate/first window that opens and asks the user to choose WHICH backup set to run?

(d) If I now make a snapshot of one of these individual partitions...
Would it start anew, or make use of a previous snapshot included in the default backup set?
e.g. /mnt/sdb1/snap2/recent.1/mnt/sda1

2. "Wouldn't you want to back up your partitions (the same ones) on a regular basis?"
(a) I don't backup to a fixed schedule.

(b) I backup quite regularly according to whether/when I think there's a need.
e.g. Perhaps I'd backup the Windows partition immediately prior to making some significant or risky change to the software. [Installing a program?]
But no need to make yet another backup of data partitions I know have not changed since the last backup.

3. "I just finished the modifications to the code to make it not use the default exclusion patterns when there is no custom exclusion pattern and the 'Add default exclusions' is unticked"
Nice! :D 8)
I uninstalled the old and installed the new.
Noticed that the old configurations were present in the new!
Did uninstalling leave in place certain files?
Not keen on that, but it was quite helpful in these circumstances.
Would it be possible for the user to be asked about this at uninstall?

4. "starting with the latest version (3.2-18), backup paths that have a custom exclusion pattern list are marked"
That's good; very helpful to see the X against the backup sources listed that have custom exclusion entries.
No X's against those with [NEW] empty custom exclusion files.
[Those with old empty custom exclusion files had X's against them until I reset them]
Attachments
00.jpg
(20.33 KiB) Downloaded 987 times

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#28 Post by Sylvander »

1.
(a) What is it that the "snap2shell" window displays?
Differences?

(b) e.g. I just updated windows, and also the Avira Anti-virus program installed within Windows, and now here in Twinkle-421 [having completed a new snapshot] I can see at the bottom of the snap2shell window, what appears to be a list of [only = nothing but] the changed files in the Firefox profile held on sda5 .

2. Would it be possible to include a scroll-bar in snap2shell so that it would be possible to browse back up through the display of folders/files listed?
Pretty much the most obvious thing missing from snap2 is the ability to view "differences".
This ability is much to be desired, because the user can discover from viewing these whether there has been any interesting/undesirable/disastrous change in the folders/files.
e.g. Vital files deleted in error.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#29 Post by lstandish »

Sylvander,
I'm glad snap2 is working out for you.

Unfortunately, the code changes to handle default exclusion patterns differently, and (previously) to remove the xterm dependency broke the code in 2 different ways! I'm very sorry, but, well, it's still 'beta'.

This affected the (1) remote authentication process, and (2) mirror-type (not snapshot) backups. So if you did only snapshot-type local backups, it didn't affect you. The fixed version is 3.2-20, download: http://files.lstandish.com/

I'll consider adding a way to choose the backup set upon execution. Currently of course you can only execute backups for the currently active set. However, the feature set is frozen now, since I want to finish testing and create a stable/tested release.

Different backup sets will not share hard link references, unless they also share the same backup storage directory, which I don't think would work for you. So the default backup set is not seen by any other sets.

You should avoid having any partitions named in more than one set. To avoid a time-consuming repeat of what you've backed-up, you can move partition backups in your default.set's backup storage directory to other storage directories. In the new storage dir, make a folder recent.1 and move the selected partition backups over. If all the backups are on the same partition, moving them will not involve actually moving the data, so it will be fast.

I too like the freedom of being able to run a backup at any time, not bound to a fixed time schedule. When I first wrote this years ago (the non-GUI shell script), that was one the gripes I had with existing snapshot backup systems. They would ONLY work when run at a certain time by cron.

snap2, on the other hand, allows the user to back up on a whim, several times per day if desired. Still the "daily", "weekly", and "monthly" backups will be automatically 'spaced out" in time, and rotated. So snap2 is a merger of the notions of sporadic backup and structured, time-based backup.

>(a) What is it that the "snap2shell" window displays?
Differences?
Yes, it lists ONLY the files that are backed-up on that run (the differences. It is recorded in the logfile in the top of each backup (for example, in directory recent.1). To see what was changed in a given backup, you would open its logfile. But I see that a button to display this file in a window would be nice. I'll add that later. That will be bit tricky since the logfile is stored with the backups on the remote server for remote backups. Hmmm, maybe it would be better to use local storage for ALL logfiles... or do both? What do you think?

Unfortunately, since I modified the code to allow rsync to create the hardlinks (fixing the problem with permissions/ownerships of older snapshots being changed to those of the newest snapshot), the logfile no longer indicates when files are deleted, only when files are backed-up. I will try increasing the verbosity level of rsync to see if that helps.

Also, I will add the scrollbar to snap2shell's terminal window. That would certainly be useful. Thanks for the suggestions!
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#30 Post by Sylvander »

1. "You should avoid having any partitions named in more than one set"
Done. :D

2. "you can move partition backups in your default.set's backup storage directory to other storage directories. In the new storage dir, make a folder recent.1 and move the selected partition backups over"
Guessed that could be done from what you'd said previously.
Hence it's already done, and it appears to have worked well. :D

3. "If all the backups are on the same partition, moving them will not involve actually moving the data, so it will be fast"
That's exactly the way it was. :D

4. " I see that a button to display this file in a window would be nice"
(a) So isn't it possible to have a scrollbar on the snap2shell window?
If not, then...

5. "That will be bit tricky since the logfile is stored with the backups on the remote server for remote backups"
Whoa, a bit too technical for me.
Does that mean a logfile held on a remote server cannot be viewed on the local machine?
Unless copies are held on the local machine?

6. "maybe it would be better to use local storage"
Seems like a bad idea to separate the log files from the backups; I like that they are held very near the backups.

7. "or do both?"
Seems less than ideal to duplicate files.

8. "the logfile no longer indicates when files are deleted, only when files are backed-up"
(a) That makes sense to me; seems like a good way to do things.
[Not so, faulty thinking by me. I want to know ALL differences]
After all, this program only makes backups, which seems good...
[Again faulty thinking, see (b) below]
No need to make it restore, so no duplication of functionality.
Restoring any chosen number of folders/files is done using other programs; again, I like that.
e.g. Xfe can/would do that quite nicely.

(b) Oops, just realized an important difference.
One of the good features in SyncBack is that it not only will alter a backup to add the "differences", but it will do the same during a restore.
i.e. Restore only the differences to make the [recently changed?] source the same as the backup [of the arrangement prior to the change].

9. What about a program that could compare 2 existing snap2 backups [e.g. sda3 recent.1 versus recent.2 (or recent.3)], and list the differences?
In Xfe I can compare 2 file-panes side-by-side, and see the obvious differences, but not a definitive list of differences.
I suppose a file-compare program could compare 2 log-files, but I'm skeptical how easily/well it would do that.

10. "I will add the scrollbar to snap2shell's terminal window"
SUPERB! :D 8)

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#31 Post by lstandish »

Sylvander,
The (optional) logfile stored with each snapshot backup indicates which files were backed up (the differences between that backup and the next oldest.) I do think it would be good to allow viewing the latest backup log (whether stored locally or on a remote machine) in a snap2 gtkdialog window.

For more in-depth information, I think the user should open the storage directory and browse through the log files in each snapshot directory.

I think these snapshot logfiles should normally give you what you want to know about a backup event. It's true they don't tell you which files were NOT backed up compared to the previous backup. As pointed out previously, to provide that information snap2 would have to create the hard links via cp rather than via rsync, and this would introduce the ownership/permissions bug (old snapshots can acquire permissions/ownerships of newer snapshots).

If you want more in-depth info about the differences between snapshots, you could run a directory comparison tool such as Krusader's "Synchronize Directories." That's non-Puppy, of course. It's really first-class.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#32 Post by lstandish »

Sylvander, I found a great graphical directory comparison utility for you (called dirdiff), and make a pet package out of it (attached). You can compare up to 5 directories at once. It needs the Puppy tk/tcl libraries. Since they are listed as dependencies, they should be installed automatically.

Here's the Debian package description:
"Dirdiff can handle up to 5 trees. It displays a main window with a
list of the files which are different between the trees, with colored
squares to indicate the relative ages of the versions. A menu allows you to display the differences between any two of the versions in another window. Another menu allows you to copy the file from one tree to another."

I think I'll put it in the snap2 download area. (For Puppy 4.3.1)
Attachments
dirdiff-2.1-4fixed2.pet
(34.14 KiB) Downloaded 1412 times
Last edited by lstandish on Mon 25 Jan 2010, 02:28, edited 1 time in total.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

2byte
Posts: 353
Joined: Mon 09 Oct 2006, 18:10

#33 Post by 2byte »

Istandish,

Can you add checking for mounted folders to snap2?
I just tried to backup an unmounted partition to another unmounted partition, it failed of course but there was no error reported.
.


Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#34 Post by Sylvander »

1. @lstandish
(a) I installed dirdiff using the pet you supplied.
At the possible end of the installation there wasn't the usual display of notifications; just saw a quick flash of more than 1 window.

(b) "Menu->Utility->dirdiff" is listed, but if I click on this nothing happens.

(c) Entered the command dirdiff in a terminal and got the response:
/usr/bin/dirdiff: line3: exec: wish: Not found

2. Any ideas how to get it working?

3. Ran "Menu->Setup->Check dependencies" on dirdiff-2.1-4, which reported:
/usr/lib/libfilecmp.so.0.0
Has missing library files:
libtcl8.5.so.0
And that Puppy had examined all user installed packages and found the following missing dependencies:
tcl tk

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#35 Post by lstandish »

Sylvander,
The problem with dirdiff is that tcl and tk need to be installed. When I built the pet I specified that tcl and tk are dependencies. Maybe I need to specify the full package name when I specify the dependency. Anyway, if you install the tcl and tk libraries, dirdiff ought to work. Search for "tcl" in the Puppy package installer program - you'll see a package with a name like tcl8.5nothread.pet (that's not exactly it - I don't have Puppy running now to check it.) The tcl package will I think require the tk package, plus a few other pet packages, totaling something over 1 meg in size. I'll try to fix the dirdif package.

2byte, I'll add a check for mounted partitions to a future version - thanks for that excellent suggestion!

Sylvander and 2byte, please make sure that you are using version 3.2-20. 3.2-19 had mirror backup broken and 3.2-18 also had SSL remote authentication broken.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#36 Post by Sylvander »

1. OK...
(a) Used PPM to install both:
tcl-8.5a4nothreads
And...
tk-8.5a4nothreads

(b) dirdiff-2.1-4 now runs and functions. :D
It's looking good so far; looks like it will provide useful info on "differences" between partition contents and backups, or between this backup and that.

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#37 Post by lstandish »

Good! Notice that dirdiff also can show color-coded differences between one file and another, so you can see exactly what changed from one version of a file to another. Plus you can apply or unapply file differences.

I made a new pet package for dirdiff giving the fill name of the dependency packages. I'll test it soon. (I made it using dir2pet running on Debian.)

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#38 Post by Sylvander »

1. "you can apply or unapply file differences"
Woa...
Could you elaborate?
e.g. Exactly what is done, and How?

User avatar
lstandish
Posts: 126
Joined: Fri 06 Jun 2008, 13:22

#39 Post by lstandish »

I just discovered this program myself, looking for a lightweight directory difference analyzer for Puppy, so I don't know much about it. However, I just did some tests, comparing a directory containing a file from snap2 (snap2engine), with a directory containing a previous version of snap2engine.

It is really cool. When I first ran dirdiff and selected the 2 directories, I saw this:
Image

This shows that both directories have the file 'snap2engine' and that the right-hand file is older than the left-hand file. Also, it shows that the left-hand dir has 'snap2' and 'snap2shell', files that are not found in the right-hand directory.

Then I double-clicked the snap2engine line, and saw this:

Image

This window shows the differences between the 2 files. The green text is the section found in the newer file that is not found in the older file. Note that I have ticked the first 3 checkboxes at the start of the lines. You do this to indicate which changes you want to work with for a subsequent "merge" operation. I clicked 'merge' in the menu and chose to "update /home/lloyd/bin". That's the older, right-hand file (pink name).

Then a window appeared to show me the result of merging my ticked changes to the older file. I scrolled down, and saw this:

Image

At this point you can save the merged files whereever you wish. I'm impressed. I really like dirdiff.

I made a new dirdiff pet and replaced the original (above).
Last edited by lstandish on Mon 25 Jan 2010, 02:26, edited 2 times in total.
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
[url]http://standish.home3.org/snap2[/url]

User avatar
8-bit
Posts: 3406
Joined: Wed 04 Apr 2007, 03:37
Location: Oregon

#40 Post by 8-bit »

Dirdiff fixed pet is not quite fixed.
You did not set the executable dirdiff as being executable
When I installed the Pet for it, and clicked on the menu entry, nothing happened.
So I tried running it from a terminal and got Permission Denied.
I then went to it's location of /usr/bin/dirdiff and set the executable properties to allow the scrip to run.
That solved it.

Post Reply