Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 26 Nov 2014, 07:24
All times are UTC - 4
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
snap2 rotating snapshot backups for Puppy
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 2 of 11 Posts_count   Goto page: Previous 1, 2, 3, 4, ..., 9, 10, 11 Next
Author Message
Sylvander

Joined: 15 Dec 2008
Posts: 3518
Location: West Lothian, Scotland, UK

PostPosted: Wed 20 Jan 2010, 05:41    Post_subject:  

I've been experimenting with snap2 as follows:

1. Made a [39.2GiB] ext3 Linux partition on my [80GB] external USB HDD, to which to save the backups.
Had to use GParted to non-destructively re-arrange partitions to achieve this.

2. Configured snap2 to backup sda1, sda3, sda5, sda6, sda7.
Backup storage on /mnt/sdb1/snap2.
Unticked "Add Default Exclusions" and clicked "Save above settings".
Despite this it seems these default exclusions are being used! Sad

3. Made a couple of snapshot backups yesterday, and another today.
38.89GiB used, and 327.86MiB free. [Whew, that was close!]

4. Deleted one of the snapshots made yesterday, with no discernible ill effect.
Snapshots have been auto-named recent.1 [newest] & recent.2 [next newest = oldest]

5. QUESTIONS
(a) Is there any way to backup only CHOSEN sources from amongst those specified/listed in snap2?

(b) Would it be possible to display "differences" as in SyncBack?

(c) Is it OK to exclude the Windows pagefile.sys?
00.jpg
 Description   
 Filesize   12.71 KB
 Viewed   1026 Time(s)

00.jpg

Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3518
Location: West Lothian, Scotland, UK

PostPosted: Wed 20 Jan 2010, 10:06    Post_subject:  

1. Completed yet another snap2 snapshot backup.

(a) Added a custom exclusion for [highlighted]->sda1 = - pagefile.sys
The result was that this file wasn't present in new recent.1 mentioned in (c) below [the backup made immediately after].

(b) Clicked on "Snapshot Backup Now".
This took 5 min to complete and reduced the remaining space from 2.8GB to 2.6GB [0.2GB used].

(c) Having deleted the recent.2 folder prior to the backup...
During this new backup...
snap2 renamed recent.1 to recent.2, and made a new recent.1->[the latest backup].
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Wed 20 Jan 2010, 11:20    Post_subject:  

Sylvander,
> Unticked "Add Default Exclusions" and clicked "Save above
> settings". Despite this it seems these default exclusions are being used!

You're right, 'Add Default Excludes' only adds (or not) the default exclude patterns to any custom exclude patterns. If there are no custom exclude patterns, then unticking 'Add Default Excludes' is ignored, and the default patterns are used. (It never occurred to me that anyone would want no exclude patterns!)

I'll change that behavior to work as expected. Meanwhile, to get rid of all exclude patterns you can create an empty custom exclude list for each specified backup path by clicking the 'Exclusions for Selected Snapshot Entry' button, then click 'Save'. By doing that, and unticking the 'Add Default Excludes', you'll have what you want.

You may want to subscribe to the 'snap2 version history' post (http://www.linuxbackups.org/node/26) to be notified when I release the next snap2 version.

a) Is there any way to backup only CHOSEN sources from amongst those specified/listed in snap2?
Do you mean, use exclude patterns in the opposite sense, so that they specify files to include rather than files to exclude? If that's what you mean, yes. See http://www.linuxbackups.org/node/31

(b) Would it be possible to display "differences" as in SyncBack?
I think that would be extremely complicated. I have no plans for doing that.

(c) Is it OK to exclude the Windows pagefile.sys?
That's the Windows virtual memory file, right? If so, I don't think you would want to back that up. I would certainly exclude it.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Wed 20 Jan 2010, 11:24    Post_subject:  

Sylvander,
> Unticked "Add Default Exclusions" and clicked "Save above
> settings". Despite this it seems these default exclusions are being used!

You're right, 'Add Default Excludes' only adds (or not) the default exclude patterns to any custom exclude patterns. If there are no custom exclude patterns, then unticking 'Add Default Excludes' is ignored, and the default patterns are used. (It never occurred to me that anyone would want no exclude patterns!)

I'll change that behavior to work as expected. Meanwhile, to get rid of all exclude patterns you can create an empty custom exclude list for each specified backup path by clicking the 'Exclusions for Selected Snapshot Entry' button, then click 'Save'. By doing that, and unticking the 'Add Default Excludes', you'll have what you want.

You may want to subscribe to the 'snap2 version history' post (http://www.linuxbackups.org/node/26) to be notified when I release the next snap2 version.

a) Is there any way to backup only CHOSEN sources from amongst those specified/listed in snap2?
Do you mean, use exclude patterns in the opposite sense, so that they specify files to include rather than files to exclude? If that's what you mean, yes. See http://www.linuxbackups.org/node/31

(b) Would it be possible to display "differences" as in SyncBack?
I think that would be extremely complicated. I have no plans for doing that.

(c) Is it OK to exclude the Windows pagefile.sys?
That's the Windows virtual memory file, right? If so, I don't think you would want to back that up. I would certainly exclude it.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3518
Location: West Lothian, Scotland, UK

PostPosted: Wed 20 Jan 2010, 15:02    Post_subject:  

1. "I'll change that behavior to work as expected"
Great! Many thanks. Very Happy

2. "By doing that, and unticking the 'Add Default Excludes', you'll have what you want"
This will do quite nicely in the meantime.
I was concerned that perhaps something useful or vital might have been excluded.

3. "Do you mean, use exclude patterns in the opposite sense, so that they specify files to include rather than files to exclude?"
(a) No, what I mean is...
Within snap2, I specify a list of partitions whose contents I MAY want to backup at this time or that...
But I probably do NOT want to backup ALL at each and every backup session.
There may be a good reason to backup one [or more] particular partitions at this time or that.
It's nice to be able to choose certain partitions [for backup] from amongst those listed.
[Hold down "Ctrl" and highlight those to be backed -up [from among the list], then click the backup button?]
Instead, the program is backing-up each and every listed partition at each and every backup session.
NO choice permitted?

(b) I can do this right now using SyncBack, and I'm trying to decide whether it would be sensible/prudent/justified to switch from using SyncBack to using snap2 for this type of job [making folder/file copies of partition contents].
To supplement the making of image backups.
Snap2 is better able [than SyncBack] to make backups of BOTH Linux AND Windows partitions.
If I don't lose too much functionality I'll switch. Very Happy
This would be an important change for me.
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Wed 20 Jan 2010, 15:48    Post_subject:  

Snap2 started as a program to do automatic non-interactive backup (via cron), and it is still oriented toward running a predefined backup job. However, you can get the result you want (fine-grained control over which partitions or sets of directories to back up) by creating additional backup 'sets'.

You could create a new backup set to backup any group of directories that you want. The result is that you back up only part of the partitions or folders. If you do this, you should however specify a different storage folder.

You can also set up a menu entry to execute a backup (via execution of snap2shell), and you can of course specify which backup set to back up, as an argument to snap2shell.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3518
Location: West Lothian, Scotland, UK

PostPosted: Thu 21 Jan 2010, 04:32    Post_subject:  

1. "you can get the result you want (fine-grained control over which partitions or sets of directories to back up) by creating additional backup 'sets'"
(a) Are there any instructions on how to do that?

(b) Seems I'd need to create various backup sets for every possible combination of partitions.
Not a very elegant solution.
I don't see myself doing that. Sad

(c) Or else configuring the program anew [to specify the partitions to backup] each time I run it.
Wouldn't want to do that either. Sad

2. "The result is that you back up only part of the partitions"
I don't think I'd ever want to backup only PARTS of chosen partitions.
I'd want to backup only certain chosen partitions, but backup ALL of the contents of those.
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Thu 21 Jan 2010, 09:55    Post_subject:  

Yes, you'd need to create a backup set for every combination of partitions you would back up. However, that's easy: A new set is created on the ADVANCED SETTINGS page by just typing in a new set name and clicking 'Change'. Then you follow the 3 step instructions at the top to configure the new backup set.

As I said, this is designed for those who do regular backups of predefined sets of directories (partitions in your case). It is really not intended for sporadic use or one-time backup. Wouldn't you want to back up your partitions (the same ones) on a regular basis?

In order to assist and encourage regular backup, snap2 will set up cron tasks to execute a backup set on certain days at a certain time.

I just finished the modifications to the code to make it not use the default exclusion patterns when there is no custom exclusion pattern and the 'Add default exclusions' is unticked.

Also, starting with the latest version (3.2-18), backup paths that have a custom exclusion pattern list are marked. That was a glaring deficiency, I think. 3.2-18 should be available a few minutes after I make this post.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2

Edited_times_total
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Thu 21 Jan 2010, 09:57    Post_subject:  

duplicate post removed
Back to top
View user's profile Send_private_message 
alienpup


Joined: 18 Oct 2009
Posts: 9
Location: Arizona, USA

PostPosted: Thu 21 Jan 2010, 20:12    Post_subject:  

lstandish wrote:
Yes, you'd need to create a backup set for every combination of partitions you would back up.

I've run into a very significant issue when attempting to backup pup_save files using any rsync-based utility. The problem is that Puppy does not "touch" the pup_save while/after updating it. Because the file's time stamp does not change, rsync does not detect that the file has changed. The pup_save is backed one time and one time only.

I've solved the problem in a backup script of my own by explicitly touching the pup_save file before running rsync.

As this problem likely exists in all Puppies and could cause other problems, it probably ought to be brought to Barry's attention.
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Thu 21 Jan 2010, 22:19    Post_subject:  

rsync decides to back up a file when it has changed in size or in last-modified time. If pupsave files don't get a fresh timestamp when Puppy updates them, I think that just about any backup utility (not just rsync) would have a problem detecting that it should be backed-up.

Your use of touch just before running a backup sounds like the best workaround until Barry fixes this.

By the way, since rsync transmits only the (compressed) differences in files when backing up, it is particularly suited to backing up big files such as pupsave files.

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3518
Location: West Lothian, Scotland, UK

PostPosted: Fri 22 Jan 2010, 05:33    Post_subject:  

1. "you'd need to create a backup set for every combination of partitions you would back up"
OK, beginning to get a feel for this. Very Happy
(a) Within the "Advanced Settings" pane...
Made an "Active Backup Set" [new name] for [almost] each individual partition [sda1+sda5 (Windows+its-data-files), sda3 [BoxPup], sda6 [music], sda7 [photos]]...

(b) After giving each new name [e.g. sda1,sda5.set], clicked the "Change" button and then completed the "3 easy steps".
See the screenshot below of the storage folders I made/used.
One folder for each partition.
Might need to eliminate the default that makes a backup of ALL partitions to the "snap2" folder.
Would that result in a time-consuming repeat of first-time backing-up of all the individual partitions?

(c) So now I can choose to backup partitions individually, and when I like.
Might it be possible [upon running the program after configurations have been completed] to make that choice more prominent?
e.g. When the user runs the "Backup Now..." from the menu...
If there's more than 1 backup set configured...
To have a separate/first window that opens and asks the user to choose WHICH backup set to run?

(d) If I now make a snapshot of one of these individual partitions...
Would it start anew, or make use of a previous snapshot included in the default backup set?
e.g. /mnt/sdb1/snap2/recent.1/mnt/sda1

2. "Wouldn't you want to back up your partitions (the same ones) on a regular basis?"
(a) I don't backup to a fixed schedule.

(b) I backup quite regularly according to whether/when I think there's a need.
e.g. Perhaps I'd backup the Windows partition immediately prior to making some significant or risky change to the software. [Installing a program?]
But no need to make yet another backup of data partitions I know have not changed since the last backup.

3. "I just finished the modifications to the code to make it not use the default exclusion patterns when there is no custom exclusion pattern and the 'Add default exclusions' is unticked"
Nice! Very Happy Cool
I uninstalled the old and installed the new.
Noticed that the old configurations were present in the new!
Did uninstalling leave in place certain files?
Not keen on that, but it was quite helpful in these circumstances.
Would it be possible for the user to be asked about this at uninstall?

4. "starting with the latest version (3.2-1Cool, backup paths that have a custom exclusion pattern list are marked"
That's good; very helpful to see the X against the backup sources listed that have custom exclusion entries.
No X's against those with [NEW] empty custom exclusion files.
[Those with old empty custom exclusion files had X's against them until I reset them]
00.jpg
 Description   
 Filesize   20.33 KB
 Viewed   881 Time(s)

00.jpg

Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3518
Location: West Lothian, Scotland, UK

PostPosted: Fri 22 Jan 2010, 09:51    Post_subject:  

1.
(a) What is it that the "snap2shell" window displays?
Differences?

(b) e.g. I just updated windows, and also the Avira Anti-virus program installed within Windows, and now here in Twinkle-421 [having completed a new snapshot] I can see at the bottom of the snap2shell window, what appears to be a list of [only = nothing but] the changed files in the Firefox profile held on sda5 [I moved it there way back when].

2. Would it be possible to include a scroll-bar in snap2shell so that it would be possible to browse back up through the display of folders/files listed?
Pretty much the most obvious thing missing from snap2 is the ability to view "differences".
This ability is much to be desired, because the user can discover from viewing these whether there has been any interesting/undesirable/disastrous change in the folders/files.
e.g. Vital files deleted in error.
Back to top
View user's profile Send_private_message 
lstandish


Joined: 06 Jun 2008
Posts: 126

PostPosted: Fri 22 Jan 2010, 11:10    Post_subject:  

Sylvander,
I'm glad snap2 is working out for you.

Unfortunately, the code changes to handle default exclusion patterns differently, and (previously) to remove the xterm dependency broke the code in 2 different ways! I'm very sorry, but, well, it's still 'beta'.

This affected the (1) remote authentication process, and (2) mirror-type (not snapshot) backups. So if you did only snapshot-type local backups, it didn't affect you. The fixed version is 3.2-20, download: http://files.lstandish.com/

I'll consider adding a way to choose the backup set upon execution. Currently of course you can only execute backups for the currently active set. However, the feature set is frozen now, since I want to finish testing and create a stable/tested release.

Different backup sets will not share hard link references, unless they also share the same backup storage directory, which I don't think would work for you. So the default backup set is not seen by any other sets.

You should avoid having any partitions named in more than one set. To avoid a time-consuming repeat of what you've backed-up, you can move partition backups in your default.set's backup storage directory to other storage directories. In the new storage dir, make a folder recent.1 and move the selected partition backups over. If all the backups are on the same partition, moving them will not involve actually moving the data, so it will be fast.

I too like the freedom of being able to run a backup at any time, not bound to a fixed time schedule. When I first wrote this years ago (the non-GUI shell script), that was one the gripes I had with existing snapshot backup systems. They would ONLY work when run at a certain time by cron.

snap2, on the other hand, allows the user to back up on a whim, several times per day if desired. Still the "daily", "weekly", and "monthly" backups will be automatically 'spaced out" in time, and rotated. So snap2 is a merger of the notions of sporadic backup and structured, time-based backup.

>(a) What is it that the "snap2shell" window displays?
Differences?
Yes, it lists ONLY the files that are backed-up on that run (the differences. It is recorded in the logfile in the top of each backup (for example, in directory recent.1). To see what was changed in a given backup, you would open its logfile. But I see that a button to display this file in a window would be nice. I'll add that later. That will be bit tricky since the logfile is stored with the backups on the remote server for remote backups. Hmmm, maybe it would be better to use local storage for ALL logfiles... or do both? What do you think?

Unfortunately, since I modified the code to allow rsync to create the hardlinks (fixing the problem with permissions/ownerships of older snapshots being changed to those of the newest snapshot), the logfile no longer indicates when files are deleted, only when files are backed-up. I will try increasing the verbosity level of rsync to see if that helps.

Also, I will add the scrollbar to snap2shell's terminal window. That would certainly be useful. Thanks for the suggestions!

_________________
--
Lloyd
snap2 rotating snapshot backups for Puppy/Debian Lenny/Ubuntu
The convenience of full backups with the speed and disk economy of incremental backups
http://standish.home3.org/snap2
Back to top
View user's profile Send_private_message 
Sylvander

Joined: 15 Dec 2008
Posts: 3518
Location: West Lothian, Scotland, UK

PostPosted: Fri 22 Jan 2010, 12:34    Post_subject:  

1. "You should avoid having any partitions named in more than one set"
Done. Very Happy

2. "you can move partition backups in your default.set's backup storage directory to other storage directories. In the new storage dir, make a folder recent.1 and move the selected partition backups over"
Guessed that could be done from what you'd said previously.
Hence it's already done, and it appears to have worked well. Very Happy

3. "If all the backups are on the same partition, moving them will not involve actually moving the data, so it will be fast"
That's exactly the way it was. Very Happy

4. " I see that a button to display this file in a window would be nice"
(a) So isn't it possible to have a scrollbar on the snap2shell window?
If not, then...

5. "That will be bit tricky since the logfile is stored with the backups on the remote server for remote backups"
Whoa, a bit too technical for me.
Does that mean a logfile held on a remote server cannot be viewed on the local machine?
Unless copies are held on the local machine?

6. "maybe it would be better to use local storage"
Seems like a bad idea to separate the log files from the backups; I like that they are held very near the backups.

7. "or do both?"
Seems less than ideal to duplicate files.

8. "the logfile no longer indicates when files are deleted, only when files are backed-up"
(a) That makes sense to me; seems like a good way to do things.
[Not so, faulty thinking by me. I want to know ALL differences]
After all, this program only makes backups, which seems good...
[Again faulty thinking, see (b) below]
No need to make it restore, so no duplication of functionality.
Restoring any chosen number of folders/files is done using other programs; again, I like that.
e.g. Xfe can/would do that quite nicely.

(b) Oops, just realized an important difference.
One of the good features in SyncBack is that it not only will alter a backup to add the "differences", but it will do the same during a restore.
i.e. Restore only the differences to make the [recently changed?] source the same as the backup [of the arrangement prior to the change].

9. What about a program that could compare 2 existing snap2 backups [e.g. sda3 recent.1 versus recent.2 (or recent.3)], and list the differences?
In Xfe I can compare 2 file-panes side-by-side, and see the obvious differences, but not a definitive list of differences.
I suppose a file-compare program could compare 2 log-files, but I'm skeptical how easily/well it would do that.

10. "I will add the scrollbar to snap2shell's terminal window"
SUPERB! Very Happy Cool
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 2 of 11 Posts_count   Goto page: Previous 1, 2, 3, 4, ..., 9, 10, 11 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Advanced Topics » Additional Software (PETs, n' stuff) » Utilities
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1269s ][ Queries: 13 (0.0104s) ][ GZIP on ]