Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 18 Dec 2014, 16:19
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
Speeding up the SnapMerge
Post new topic   Reply to topic View previous topic :: View next topic
Page 3 of 7 [92 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Author Message
jamesbond

Joined: 26 Feb 2007
Posts: 2232
Location: The Blue Marble

PostPosted: Mon 07 Feb 2011, 22:43    Post subject:  

Q5sys wrote:
jamesbond wrote:
Tadaa ... s9 (version 9 of the script). Same performance as s7. Comment out the echo to reduce verbosity. This code is only for the copy-down only - I'm not sure what else snapmergepuppy does. Perhaps I'll try it out later - but meanwhile, anyone is welcome to try it.

How much faster do you estimate this is over the default way of doing things? I'm quite impressed by everyones work in this thread.
for everyone. Razz

Until this is really merged into a puplet for testing, no one can tell for sure, unfortunately. Benchmarks doesn't always translate into real-world performance Embarassed

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Mon 07 Feb 2011, 22:51    Post subject:  

Umm... are we trying to manually do what aubrsync does?
http://aufs.sourceforge.net/aufs2/brsync/README.txt

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2232
Location: The Blue Marble

PostPosted: Mon 07 Feb 2011, 23:18    Post subject:    

technosaurus wrote:
Umm... are we trying to manually do what aubrsync does?
http://aufs.sourceforge.net/aufs2/brsync/README.txt
Hahaha yes !!! Good find technosaurus Smile

EDIT: Incidentally that script also use rsync ... so we're in the right track (when trying to re-invent the wheel, that is) Very Happy

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Mon 07 Feb 2011, 23:29    Post subject:  

I really have no clue how the save to cd/dvd parts work, but...
http://freshmeat.net/projects/rdiff-backup/
Seems like it could be sensible?

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Mon 07 Feb 2011, 23:40    Post subject:  

Here is the actual script.
aubrsync.gz
Description 
gz

 Download 
Filename  aubrsync.gz 
Filesize  3.2 KB 
Downloaded  286 Time(s) 
Back to top
View user's profile Send private message Visit poster's website 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Mon 07 Feb 2011, 23:50    Post subject:  

Looking at comments in the code.

Quote:
The dst_branch must be mounted as writable.
During the operation, the mntpnt is set readonly.
If you are opening a file for writing on the writable branch,
you need to close the file before invoking this script.


They do have a move option, but it has problems.

Quote:
Like above (2 branches), move and reflect all modifications
from upper to lower. Almost all files on the upper branch will
be removed. You can still use this aufs after the
operation. But the inode number may be changed. If your
application which depends upon the inode number was running at
that time, it may not work correctly.


I get the feeling this is not going to be very transparent to the user. Smile
Back to top
View user's profile Send private message Visit poster's website 
jamesbond

Joined: 26 Feb 2007
Posts: 2232
Location: The Blue Marble

PostPosted: Tue 08 Feb 2011, 07:20    Post subject:  

Long way ahead ... but, I think, if one doesn't really try to deliberately break things (like removing /lib or /usr/sbin and then re-creating and re-populating them), it should be ok for most of them. Again, it's a statement that can only be proven by experiments ...
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2232
Location: The Blue Marble

PostPosted: Tue 08 Feb 2011, 09:57    Post subject:  

Took a jump and tried my script (replacing snapmergepuppy directly). Run the script - doesn't crash, but all other executables are gone (and freemem shows zero space available). But it doesn't crash (although reboot is impossible - power button is required).
After a reboot, I removed all the tmpfs-deletion stuff - and things work! (doesn't crash, system continue as normal, and data is saved, reboot works properly). I think I just need to be careful on what can and cannot be deleted, as jemimah said before.
I tried this on my netbook with harddisk, so the delay isn't noticeable. I should try this on my eeepc with the slow sd-card access - and then we can get a benchmark.

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Tue 08 Feb 2011, 11:55    Post subject:  

So the slow performance of the script is really only a problem on shutdown. However while you are running, it'd actually be better if the snapmerge doesn't hog your cpu.

It should be fine to use the aubrsync at shutdown as all files should be closed at that point anyway.

I've been stress testing Dougal's patch and it's about twice as fast. But I've discovered some bugs during my tests - deleted files coming back from the dead after a reboot. But turns out the original script has the same problem - hopefully I can identify the source.
Back to top
View user's profile Send private message Visit poster's website 
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Tue 08 Feb 2011, 15:37    Post subject:  

jemimah wrote:
But turns out the original script has the same problem - hopefully I can identify the source.

I see debugging commented out "set -x"
Back to top
View user's profile Send private message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Tue 08 Feb 2011, 16:26    Post subject:  

The first major problem is that the script is copying down files without checking if the whiteout is there in the base layer. So that is the probable source for the I/O errors I've been seeing occaisonally.
Back to top
View user's profile Send private message Visit poster's website 
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Tue 08 Feb 2011, 19:14    Post subject:  

Man - I wish I had looked at this stuff before starting the zdrv cutter. It seems like it may be possible to write the needed modules directly to the save file. (I could probably just "touch" those and then delete the zdrv altogether without having to do all of the crazy shifting around and rebuilding the squash file) oh well - hindsight is 2010
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2232
Location: The Blue Marble

PostPosted: Tue 08 Feb 2011, 20:57    Post subject:  

Applied my s10 script (same as s9 except I optimise the "find" command, using the same command as found in the original snapmergepuppy, and add --inplace to rsync) to my puppeee running on 701SD with a very slow SD card.
Merge time is reduced frrom 30secs to less than 3secs. No data loss, no I/O error so far. No ghost file coming back after deletion either Smile This is on a 128MB save file.
Oh yes I disable the last rm command that clears the tmpfs - change that to echo. Haven't got around to make it work.

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Tue 08 Feb 2011, 21:18    Post subject:  

Why is it that files with a whiteout in the TMPFS layer also appear in lower layers?

example:
/initrd/pup_rw/etc/.wh.windowmanager.openbox
/initrd/pup_ro2/etc/windowmanager.openbox

Edit Oh..got it....File is in lupu.sfs and got deleted..

To be safe, looks like aubrsynch needs to be run during shutdown (if I understand it correctly).
Back to top
View user's profile Send private message 
jemimah


Joined: 26 Aug 2009
Posts: 4309
Location: Tampa, FL

PostPosted: Tue 08 Feb 2011, 23:21    Post subject:  

jamesbond wrote:
Applied my s10 script (same as s9 except I optimise the "find" command, using the same command as found in the original snapmergepuppy, and add --inplace to rsync) to my puppeee running on 701SD with a very slow SD card.
Merge time is reduced frrom 30secs to less than 3secs. No data loss, no I/O error so far. No ghost file coming back after deletion either Smile This is on a 128MB save file.
Oh yes I disable the last rm command that clears the tmpfs - change that to echo. Haven't got around to make it work.


Does rsync stop copying before your file system fills up? I guess the worry is corruption if you run out of space in the middle of a transfer.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 3 of 7 [92 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0911s ][ Queries: 12 (0.0163s) ][ GZIP on ]