Yes, _1_ is init. I used to have the code $2 > 2 and $3 > 2 (don't pause init and any of the [kthreads], or any processes started by them) - but pausing them seems to have no harm, I still have my files all saved, and it gives me the warm feeling that no one can spawn a process behind my back - so that's why they're included also.technosaurus wrote:I'm not on a puppy box at the moment, but a little concerned about pausing init and subsequently running processes before unpausing it (not sure if it matters though) - maybe add a ... && $2 != "1" ... if there are problems (1 _is_ the init procsess right?)
Speeding up the SnapMerge
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
Yes, probably should replace that with "expr substr $LEAF 4 500" or something like it. Or change that shell to /bin/bash. That's one of my gripe about ash - for bash, you can go to http://www.gnu.org/software/bash/manual/bashref.html and see what you can and can't do with it. There is nothing like it for ash, the closest I can find is http://linux.die.net/man/1/ash - but then busybox ash (as you said, depending on which version) can do a little bit more than that, and I can't find any documentation on busybox ash (except reading the source perhaps) - what gives?Dougal wrote:It's version-dependent. My version of Busybox Ash doesn't support it. Busybox also supports "let", which is a bashism, so not safe to rely on.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
By that logic we should only use /bin/bash if it is bash 2.x compatible ... which it probably is not ... just include a note that you may need to upgradeDougal wrote:It's version-dependent. My version of Busybox Ash doesn't support it. Busybox also supports "let", which is a bashism, so not safe to rely on.jamesbond wrote:In my system (and also Fluppy), /bin/ash is symlinked to busybox. Busybox ash seems to support this:It isn't documented, and if you do have a link to where busybox ash is documented, I'm happy to take it on (busybox applet man page list the details of every other applet except ash).Code: Select all
# busybox ash # a=abcde # echo ${a:3} de # exit
AFAIK the only ash documentation is in the source tree.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].
Or you can exclude that from being touched. Editing awk file is easier than recompiling I guess.jemimah wrote:In Frisbee I'm using SIGCONT to communicate with the tray. I guess if this ends up working I should probably switch it to some other signal.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
A few crash and hangups later on the real eeepc - here comes s14. Changes:
1. wrap the whole snapmerge within freeze/unfreeze
2. don't kill init and its direct descendants (otherwise hangups in certain cases - as technosaurus said)
3. aufs remount at end of merge, apparently to force re-eval of all files (otherwise some files will be missing). I'm suspicous of this, I have to do it otherwise some weird things will happen (This validates jemimah's concern earlier). As I said, my earlier attempt with remount was disastrous. So let's hope there is no bad side effect to this.
so far, so good. I did this (and its variation, deletion etc) a couple of times when testing:
cheers!
EDIT: file removed, buggy, please use latest version
1. wrap the whole snapmerge within freeze/unfreeze
2. don't kill init and its direct descendants (otherwise hangups in certain cases - as technosaurus said)
3. aufs remount at end of merge, apparently to force re-eval of all files (otherwise some files will be missing). I'm suspicous of this, I have to do it otherwise some weird things will happen (This validates jemimah's concern earlier). As I said, my earlier attempt with remount was disastrous. So let's hope there is no bad side effect to this.
so far, so good. I did this (and its variation, deletion etc) a couple of times when testing:
Code: Select all
(for a in {1..10000}; do echo file$a > file$a; done) & snapmergepuppy
EDIT: file removed, buggy, please use latest version
Last edited by jamesbond on Sat 12 Feb 2011, 13:09, edited 1 time in total.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
Ok I finally got to testing this. I'm really impressed so far! Saving is practically instant. I think this should be posted somewhere more visible to get more testers.
----
However opaque directories are not working. I test this in Puppeee by switching to openbox and editing /usr/local/bin/rox-menu. I change the 3rd line to
Then I delete an application from /usr/local/applications and run fixmenus. Then I run snapmergepuppy. Then I reboot and see if the application is back in the menu or not.
----
However opaque directories are not working. I test this in Puppeee by switching to openbox and editing /usr/local/bin/rox-menu. I change the 3rd line to
Code: Select all
PUPMODE=12
Great ! Perhaps one of the mods can re-assign this thread to "Cutting edge" or something?jemimah wrote:Ok I finally got to testing this. I'm really impressed so far! Saving is practically instant. I think this should be posted somewhere more visible to get more testers.
----
I tested under Fluppy 12. I moved Xinvaders3d from /usr/share/applications to /root, re-run fix menus, and then run snapmergepuppy. I check the menu (I run openbox with rox panel) - xinvaders is now gone, so far so good. I reboot. I checked the menu - xinvaders come backHowever opaque directories are not working. I test this in Puppeee by switching to openbox and editing /usr/local/bin/rox-menu. I change the 3rd line to
Then I delete an application from /usr/local/applications and run fixmenus. Then I run snapmergepuppy. Then I reboot and see if the application is back in the menu or not.Code: Select all
PUPMODE=12
BUT, I checked
- usr/share/applications - no more xinvaders, it's gone.
- /root/my-roxapps/Menu-Items - no more xinvaders entry.
- /initrd/pup_ro1/usr/share/applications - I see whiteout file
- /initrd/pup_ro1/root/my-roxapps/Menu-Items - I also see the whiteout file
So, as far as file deletion is concerned, it seems to work fine - deleted items are deleted. Where else could rox-panel get the menu entry from? I would need to check that.
EDIT: Found the bug, working on a solution.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
I hope I've nailed this dir-opaque thing. It's very tricky guessing how it works - especially without reading aufs source in full.
s16 - combined all those scripts into one big bash script.
hopefully I've tackled all those "ghost" files that keeps coming back after reboot.
Let me know how you fare.
I'm going to remove all the previous uploaded files as they're all buggy so far.
cheers!
s16 - combined all those scripts into one big bash script.
hopefully I've tackled all those "ghost" files that keeps coming back after reboot.
Let me know how you fare.
I'm going to remove all the previous uploaded files as they're all buggy so far.
cheers!
Last edited by jamesbond on Tue 01 Mar 2011, 11:07, edited 1 time in total.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
Wow
tried s-16 in XOpup that traditionally had issues with snapmergepuppy.
No problems at all (yet). And this thing is fast!
Well done.
Now if someone can make puppy to umount the savelayer cleanly too...
tried s-16 in XOpup that traditionally had issues with snapmergepuppy.
No problems at all (yet). And this thing is fast!
Well done.
Now if someone can make puppy to umount the savelayer cleanly too...
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==
The opaque directory thing does seem to be fixed.
I copied several hundred MB of data to RAM and ran the snap merge. It took several minutes to copy it down to the usb drive. Of course I couldn't even move the mouse the whole time.
Did you move the suspend to happen before the copy down?
I guess that's a good trade for getting my RAM back but I could see how it could be very annoying if it happened at the wrong time - like during a skype call or a presentation or something.
I think we need a way to warn the user before locking up the system for more than a second or two. Or have saving and purging RAM be separate things and the purge only happens if the user requests it.
---
Mavrothol, check out this thread.
http://murga-linux.com/puppy/viewtopic.php?t=55823
I copied several hundred MB of data to RAM and ran the snap merge. It took several minutes to copy it down to the usb drive. Of course I couldn't even move the mouse the whole time.
Did you move the suspend to happen before the copy down?
I guess that's a good trade for getting my RAM back but I could see how it could be very annoying if it happened at the wrong time - like during a skype call or a presentation or something.
I think we need a way to warn the user before locking up the system for more than a second or two. Or have saving and purging RAM be separate things and the purge only happens if the user requests it.
---
Mavrothol, check out this thread.
http://murga-linux.com/puppy/viewtopic.php?t=55823
Yes, I now freeze the whole thing before deletion - I was considering your C compiler scenario, where a process may create and delete many files continuously (e.g. when updating a previous build), and I thought it may be better to freeze the whole thing.
I can revert it back to the way it was before (around s12 or so), block only during deletion.
The following is just random thoughts:
- We need to pinpoint the source of the delay - is it the rsync, or is it the other stuff (whiteouts checking, or tmpfs deletion?).
- It's easy to separate copydown and deletion, but how can we integrate back to puppy infrastructure? I prefer not touch anything outside snapmergepuppy - I'm hoping that this script can be a dropdown replacement.
- Alternatively I can let X process to continue to run though I think it may open another can of worms ...
- Putting up an Xdialog before freezing may be a good idea also. The thing is, if you only do usual stuff (browsing, edit a few documents etc) - the volume of data to be written will be small and the delay will be relatively unnoticeable. So a big red "danger" flag before every save operation may not be warranted as well (or perhaps we can just do a du -sh /initrd/pup_rw and if the value is larger than a couple of tens MB, we should pop up a warning - what do you think?)
Let me know your thoughts. Meanwhile, I'll just modularise the script even more so that it's easier to pick and choose what we want to do.
cheers!
I can revert it back to the way it was before (around s12 or so), block only during deletion.
The following is just random thoughts:
- We need to pinpoint the source of the delay - is it the rsync, or is it the other stuff (whiteouts checking, or tmpfs deletion?).
- It's easy to separate copydown and deletion, but how can we integrate back to puppy infrastructure? I prefer not touch anything outside snapmergepuppy - I'm hoping that this script can be a dropdown replacement.
- Alternatively I can let X process to continue to run though I think it may open another can of worms ...
- Putting up an Xdialog before freezing may be a good idea also. The thing is, if you only do usual stuff (browsing, edit a few documents etc) - the volume of data to be written will be small and the delay will be relatively unnoticeable. So a big red "danger" flag before every save operation may not be warranted as well (or perhaps we can just do a du -sh /initrd/pup_rw and if the value is larger than a couple of tens MB, we should pop up a warning - what do you think?)
Let me know your thoughts. Meanwhile, I'll just modularise the script even more so that it's easier to pick and choose what we want to do.
cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
I think the delay is mainly that it is just slow to copy a lot of data to a usb drive. I was thinking of the du warning too.
Of course if we're not using a lot of RAM, there's little point in deleting so maybe just skip it if we don't need it. Perhaps just detect if we're getting low on RAM and pop up the dialog offerring to purge.
The size of the brower cache probably defaults to 100MB or something and if the user is downloading things, it's not hard to amass large files.
Of course if we're not using a lot of RAM, there's little point in deleting so maybe just skip it if we don't need it. Perhaps just detect if we're getting low on RAM and pop up the dialog offerring to purge.
The size of the brower cache probably defaults to 100MB or something and if the user is downloading things, it's not hard to amass large files.
Didn't incorporate your suggestion yet - I was working on it, finished it and then see your post.
Anyway - here is s17 - modularised version of s16.
Changes -
1) warn user based on size of tmpfs and number of files in tmpfs (I included both because I'm not sure which steps causes the slow down)
2) modularised the functions
It still works as before (ie pauses the whole process), but I've separated the freeze function, snapmerge function, and cleanup function. It will be easy to change so that it will only freeze during cleanup. Look at line 204 - just move the freeze_processes in-between snapmerge and cleanup.
If we can confirm that the slowdown is because of tmpfs deletion, then there's not point in doing this. But if the slowdown is because rsync - ok perhaps it's worthwhile to take additional risks for shorter freeze time.
Let me know.
Oh yes the warn_user limit is arbitrary - it really depends on how slow things are. I'll take your feedback on this. Currently I set the limit to 64MB and 1000 files.
cheers!
Anyway - here is s17 - modularised version of s16.
Changes -
1) warn user based on size of tmpfs and number of files in tmpfs (I included both because I'm not sure which steps causes the slow down)
2) modularised the functions
It still works as before (ie pauses the whole process), but I've separated the freeze function, snapmerge function, and cleanup function. It will be easy to change so that it will only freeze during cleanup. Look at line 204 - just move the freeze_processes in-between snapmerge and cleanup.
If we can confirm that the slowdown is because of tmpfs deletion, then there's not point in doing this. But if the slowdown is because rsync - ok perhaps it's worthwhile to take additional risks for shorter freeze time.
Let me know.
Oh yes the warn_user limit is arbitrary - it really depends on how slow things are. I'll take your feedback on this. Currently I set the limit to 64MB and 1000 files.
cheers!
Last edited by jamesbond on Tue 01 Mar 2011, 11:07, edited 1 time in total.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
Thanks.mavrothal wrote:Wow
tried s-16 in XOpup that traditionally had issues with snapmergepuppy.
No problems at all (yet). And this thing is fast!
Well done.
I thought Patriot has done this, and I'm sure that at least Fluppy, later version of Puppeee and Fatdog has this included. I'm not sure whether it makes it to the base puppy (woof) though.Now if someone can make puppy to umount the savelayer cleanly too...
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
Hi Jamesbond
I tried s17 on my xo-1 olpc laptop and it does make a big speed difference. No problem so far after installing a couple of pets and deleting a few files.
Note that I am not having mavrothal's issue of devices not unmounting. Patriot's code is indeed in the rc.shutdown script. These are peculiar machines so it could be anything.
Cheers
I tried s17 on my xo-1 olpc laptop and it does make a big speed difference. No problem so far after installing a couple of pets and deleting a few files.
Note that I am not having mavrothal's issue of devices not unmounting. Patriot's code is indeed in the rc.shutdown script. These are peculiar machines so it could be anything.
Cheers
Puppy Linux Blog - contact me for access
The code is there (as Micko said) but it does a lazy umount. So if you have an ext3/4 formatted media you see that every time it recovers the journal (even if you add some time after the lazy umount for things to be written).jamesbond wrote:I thought Patriot has done this, and I'm sure that at least Fluppy, later version of Puppeee and Fatdog has this included. I'm not sure whether it makes it to the base puppy (woof) though.Now if someone can make puppy to umount the savelayer cleanly too...
I think Mick is using vfat, ext2 media thus does not see the problem, since as far as puppy is concerned the fsckme.err/.flg is removed already.
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==
Mplayer conflict?
Tried s-17 (in XOpup).
I do not know if this a bug or a feature but when gnome-mplayer is open snapmergepuppy freezes in the initial "merge requested" dialog.
Even if mplayer is paused or stopped or even without a file loaded.
Resumes when mplayer closes.
There is nothing in messages or xerrs.log indicative of a problem other than a process can not be found to be stopped and then restarted, at the "kill -STOP ${allprocess[$i]}" and "kill -CONT $i" step respectively. But this message has nothing to do with gnome-mplayer running or not.
For comparison, running a flash movie in the browser pauses no problem in snapmerge (and only as small freeze in flash).
I do not know if this a bug or a feature but when gnome-mplayer is open snapmergepuppy freezes in the initial "merge requested" dialog.
Even if mplayer is paused or stopped or even without a file loaded.
Resumes when mplayer closes.
There is nothing in messages or xerrs.log indicative of a problem other than a process can not be found to be stopped and then restarted, at the "kill -STOP ${allprocess[$i]}" and "kill -CONT $i" step respectively. But this message has nothing to do with gnome-mplayer running or not.
For comparison, running a flash movie in the browser pauses no problem in snapmerge (and only as small freeze in flash).
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==
You probably have an old version of the shutdown script.mavrothal wrote:The code is there (as Micko said) but it does a lazy umount. So if you have an ext3/4 formatted media you see that every time it recovers the journal (even if you add some time after the lazy umount for things to be written).jamesbond wrote:I thought Patriot has done this, and I'm sure that at least Fluppy, later version of Puppeee and Fatdog has this included. I'm not sure whether it makes it to the base puppy (woof) though.Now if someone can make puppy to umount the savelayer cleanly too...
I think Mick is using vfat, ext2 media thus does not see the problem, since as far as puppy is concerned the fsckme.err/.flg is removed already.
Patriot's code looks like this and does work:
Code: Select all
if [ "$SAVE_LAYER" ];then
sync
SAVEDEV="`mount | grep "/initrd${SAVE_LAYER}" | cut -f 1 -d ' '`"
SAVEFS="`mount | grep "/initrd${SAVE_LAYER}" | cut -f 5 -d ' '`"
#100615 Patriot: suggested this code to enable save-layer to remount ro...
uniFS=$(awk '/unionfs/ {print $3}' /proc/mounts) #ex: aufs
if [ "$uniFS" == "aufs" -a "$SAVE_LAYER" == "/pup_rw" ]; then
#i think only work if prepended dir is a separate f.s...
if [ "`mount | grep '^tmpfs on /tmp '`" != "" ];then #created in initrd.
mkdir -p /tmp/unrootfs
busybox mount -o remount,prepend:/tmp/unrootfs,xino=/tmp/unrootfs/xino -t $uniFS / /
sync
fi
fi
busybox mount -t $SAVEFS -o remount,ro $SAVEDEV /initrd${SAVE_LAYER} 2>/dev/null
umount-FULL -i -n -l /initrd${SAVE_LAYER} 2>/dev/null #-l is lazy unmount.
fi
I'm going to start a new save file and try to run in pupmode 13 for a few weeks and see how it goes.jamesbond wrote:Didn't incorporate your suggestion yet - I was working on it, finished it and then see your post.
Anyway - here is s17 - modularised version of s16.
Changes -
1) warn user based on size of tmpfs and number of files in tmpfs (I included both because I'm not sure which steps causes the slow down)
2) modularised the functions
It still works as before (ie pauses the whole process), but I've separated the freeze function, snapmerge function, and cleanup function. It will be easy to change so that it will only freeze during cleanup. Look at line 204 - just move the freeze_processes in-between snapmerge and cleanup.
If we can confirm that the slowdown is because of tmpfs deletion, then there's not point in doing this. But if the slowdown is because rsync - ok perhaps it's worthwhile to take additional risks for shorter freeze time.
Let me know.
Oh yes the warn_user limit is arbitrary - it really depends on how slow things are. I'll take your feedback on this. Currently I set the limit to 64MB and 1000 files.
cheers!
[url]http://saluki-linux.com[/url]