clean_whiteouts_daemon: A Daemon to Clean Whiteout File

Stuff that has yet to be sorted into a category.
Post Reply
Message
Author
User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

clean_whiteouts_daemon: A Daemon to Clean Whiteout File

#1 Post by Dougal »

I noticed that yet another person was hurt by the whiteout files resulting from Opera's cache activity, so I figured: "someone should add the cleanup once and for all to the Puppy shutdown script".
But then I thought, "But why do we need to reboot for them to be cleaned? This isn't Windows, right?"

So I wrote this daemon, to take care of the whiteout files.

What it does is sleep for some time (default is 30 minutes), then wake up and go cleaning. Then go to sleep again. And so on.
It doesn't just delete whatever it finds -- it makes sure they don't cover files in any union layer.

By default it checks all the filesystem for whiteout files, but if you want it to only look at some directories (e.g. /root), you have a config option.
There's also an option to have /tmp cleaned (since a lot of whiteout files can be created there when compiling), but I think Barry reinstated making /tmp a tmpfs, so it might not be necessary.
You can make it verbose and it will list every file it deletes in the log file (/var/log/clean_whiteouts.log).

There are some more config options, all explained in the config file (/etc/clean_whiteouts.conf).

In case anybody is worried about performance (i.e. it bloating Puppy), I don't think that should be much of a problem, as the script renices itself so it runs at a low priority and should not steal the cpu when other apps want it.

It comes with a startup file in /etc/init.d, so it will be started at bootup and stopped at shutdown (the dotpet will also start it when it installs).

It should also help make shutdown faster for people running off flash drives (PUPMODE 3/7/13), as snapmergepuppy will have less files to handle...

I haven't tested it much, but similar code was in the shutdown script for 2.14R... maybe people should test it a bit (keeping an eye on the log file) and report here if it works ok or not.

(In case anyone wonders why the script name is strangely shorter that it's title, that's because killall is stupid and can't handle long names...)
Last edited by Dougal on Sun 12 Apr 2009, 09:32, edited 1 time in total.
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

Re: clean_whiteouts_daemon: A Daemon to Clean Whiteout File

#2 Post by WhoDo »

Dougal wrote:So I wrote this daemon, to take care of the whiteout files.
Looks like a must-have feature for Puppy, Dougal. 8) I'll watch the testing with interest and if there aren't any major stumbling blocks then I'll add it to Puppy-4.2RC2

Thanks, mate.
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

#3 Post by MU »

I think this will just be needed in old Puppys up to 4.0 alpha.

They used the unionfs.ko module as unionfs driver to set up the layered filesystem.

Newer versions instead use aufs.ko, an alternative, better unionfs driver.
That one should no longer have that issue.

Or just for clarification, where there reports for the new versions of Puppy, too? (4.1 and newer)
Mark
[url=http://murga-linux.com/puppy/viewtopic.php?p=173456#173456]my recommended links[/url]

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

#4 Post by WhoDo »

MU wrote:I think this will just be needed in old Puppys up to 4.0 alpha.

They used the unionfs.ko module as unionfs driver to set up the layered filesystem.

Newer versions instead use aufs.ko, an alternative, better unionfs driver.
That one should no longer have that issue.

Or just for clarification, where there reports for the new versions of Puppy, too? (4.1 and newer)
I'm not aware of any reports of the problem where aufs.ko is used by default, Mark. What about the option to use unionfs.ko in later Puppy's with a boot parameter? Also doesn't the boot revert to unionfs.ko if aufs.ko is not available for some reason? In these cases, wouldn't Dougal's daemon provide additional fault protection? :? Maybe we could include it and have it enabled manually rather than automatically, so giving users the option. :idea:

These are just my thoughts on the subject. If you guys tell me it isn't needed at all for 4.2 with aufs.ko then I'll just forget about it. :)
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

#5 Post by Dougal »

MU wrote:I think this will just be needed in old Puppys up to 4.0 alpha.

They used the unionfs.ko module as unionfs driver to set up the layered filesystem.

Newer versions instead use aufs.ko, an alternative, better unionfs driver.
That one should no longer have that issue.
I think the newer versions use aufs, but you have a boot option (layerfs) to use unionfs.

I myself have been using unionfs, since I compiled my own kernel and the unionfs patch adds the module to the kernel tree, whereas the aufs patch just makes some changes to the kernel and expect you to compile aufs.ko after you've booted the new kernel...
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

#6 Post by MU »

I see, so your utility really is important.
The whiteout files can completely destroy a savefile, this drove me crazy in Muppy, when we did not use aufs.ko yet.

Mark
[url=http://murga-linux.com/puppy/viewtopic.php?p=173456#173456]my recommended links[/url]

User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

#7 Post by Dougal »

Before anyone goes adding this to any kind of release, let me first make a little improvements to it, so that it will only run when using unionfs -- no need for sufs or a full install...
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

User avatar
WhoDo
Posts: 4428
Joined: Wed 12 Jul 2006, 01:58
Location: Lake Macquarie NSW Australia

#8 Post by WhoDo »

Dougal wrote:Before anyone goes adding this to any kind of release, let me first make a little improvements to it, so that it will only run when using unionfs -- no need for sufs or a full install...
Done. 8)
[i]Actions speak louder than words ... and they usually work when words don't![/i]
SIP:whodo@proxy01.sipphone.com; whodo@realsip.com

big_bass
Posts: 1740
Joined: Mon 13 Aug 2007, 12:21

#9 Post by big_bass »

Dougal

I am testing your pet I am doing an experimental USB build of
woof and I need all the extra space I can get

I am multi booting 5 OS's so I ran out of room on my hard drive
for the required 10 GB of space needed
so I do need to test your cleaner


a side note:
and just something I came across looking over woof
the new woof is handing it like this
in this script
/root/woof-alpha2/boot/initrd-tree0/init


there is this comment
#w003 /usr/sbin/snapmergepuppy now saves whiteout files to save-layer, change 'ro' to 'ro+wh'.

can that be of use to you ?



big_bass

disciple
Posts: 6984
Joined: Sun 21 May 2006, 01:46
Location: Auckland, New Zealand

#10 Post by disciple »

only run when using unionfs
Even if you don't have the unionfs problem, the large number of whiteout files that can accumulate can be a little annoying :)
Do you know a good gtkdialog program? Please post a link here

Classic Puppy quotes

ROOT FOREVER
GTK2 FOREVER

Post Reply