How does Puppy do 'white out' files? (Solved)

[nic007]

Which script/scripts directs how Puppy deals with white out files? In other words - there must be a script that looks for white out files (probably in the savefile/folder) and then "mask" (hides) the files from view if "deleted" from the base sfs.

[mfb]

Hi nic007,

Late last year I tested your adrv code which worked superbly well and unbelievably efficiently, except, as you yourself wrote, it did not cater for deletions.

Hopefully, a useful answer to your post immediately above, would enable you to fix this show-stopping defect to your (otherwise) better than sliced-bread improvement.

I made very minor adjustments to your code (resized @426B) where it took 6 seconds to back-up by renaming and make a new <30M adrv (whereas my normal save sfs is 200M with some 90M used).

I also used a save sfs of 1M which never needed amending or updating ( though I don't remember if or why I needed to do that) and I also made some minor tweaks in my /root/Startup directory to fix the odd tray icon hitch.

[nic007]

Exactly the reason I'm asking this.

[rufwoof]

system code ... union/aufs mounting basically merges two folders into one, typically via allocating two folders by name, one ro, the other rw that are to be merged along with a third folder that reflects the image of the merging of those two folders. If a file somefile.txt in the ro layer is to be 'deleted' then the system creates a .wh.somefile.txt file in the rw layer to indicate that the top layer doesn't 'show' that somefile.txt file.

In short, handled by the OS, not by a script.

[nic007]

system code ... union/aufs mounting basically merges two folders into one, typically via allocating two folders by name, one ro, the other rw that are to be merged along with a third folder that reflects the image of the merging of those two folders. If a file somefile.txt in the ro layer is to be 'deleted' then the system creates a .wh.somefile.txt file in the rw layer to indicate that the top layer doesn't 'show' that somefile.txt file.

In short, handled by the OS, not by a script.

You are describing in short how the layered fs works. However, this is not my problem. When you have a frugal install on HDD any changes to the system are saved to the savefile/folder which loads on top of the base sfs as you mentioned . White out files are very clearly marked in the savefile. However, I do not work with a savefile but save my system changes to an adrv which also loads on top of the base sfs like a savefile.
In this case however, white out files are not dealt with, why? So in short - we have a situation where white out files in the savefile (loaded as /initrd/pup_rw) are dealt with correctly but not when those white out files are indicated in the adrv (loaded as /initrd/pup_a). My guess is that it has something to do with the init script in initrd.gz and I want to isolate and edit that part so that it includes the adrv as well as other sfs's that may have preference to the base sfs at load time.

[ITSMERSH]

I'm not sure about this, though I doubt it is in the init script. I think this happens after performing the switch-root when /etc/profile, rc.sysinit and such stuff is executed.

[greengeek]

I can't explain it very well but i had similar questions when I tried to make a .pet that contained the contents of pup-rw

My plan was to save that .pet and load it on top of the base sfs next boot. It didn't seem to be handling whiteouts properly. Sorry i never found an answer - but I will be watching this thread with interest.

I kind of had the feeling that there was an inbuilt "permissions" error that stopped the whiteouts being copied. I didnt finish my researching it tho'

[nic007]

I can't explain it very well but i had similar questions when I tried to make a .pet that contained the contents of pup-rw

My plan was to save that .pet and load it on top of the base sfs next boot. It didn't seem to be handling whiteouts properly. Sorry i never found an answer - but I will be watching this thread with interest.

I kind of had the feeling that there was an inbuilt "permissions" error that stopped the whiteouts being copied. I didnt finish my researching it tho'

I think there is a section in the snapmergepuppy and/or shutdown scripts that deals with white out files. We need to do a test - copy the contents of a savefile (which has white-out files) and load it as an adrv. If the white out files are then dealt with correctly, one can take it from there...

[greengeek]

No idea what value this might be - but /etc/rc.d/rc.shutdown contains the following references to .wh files:

#120522 precise puppy with aufs, have two dirs .wh..wh.orph, .wh..wh.plnk, in /initrd/pup_rw, filter out.

#120522 testing precise puppy with aufs, have two dirs here .wh..wh.orph, .wh..wh.plnk, filter out...
RDIRS="`find /initrd/pup_rw/ -mindepth 1 -maxdepth 1 -mount -type d | grep -v '/\.wh\.' | grep -v -E '/$|/mnt|/tmp|/proc|/sys|/dev|/lost' | tr '\n' ' '`"

Just posted it for reference. Why would such files only be filtered out in Precise? And why not mentioned for other pups?

Also, the file /usr/sbin/snapmergepuppy contains the following references to ".wh":

#w003 bug fix for .wh.__dir_opaque whiteout file.

#110212 recent aufs: .wh.__dir_opaque name changed to .wh..wh..opq.

#Handle Whiteouts...
find . -mount \( -regex '.*/\.wh\.[^/]*' -type f \) | sed -e 's/\.\///;s/\.wh\.//' |
while read N
do
BN="`basename "$N"`"
DN="`dirname "$N"`"
[ "$BN" = ".wh.aufs" ] && continue #w003 aufs has file .wh..wh.aufs in /initrd/pup_rw.
[ "$DN" = "." ] && continue
#110212 unionfs and early aufs: '.wh.__dir_opaque' marks ignore all contents in lower layers...
if [ "$BN" = "__dir_opaque" ];then #w003
#'.wh.__dir_opaque' marks ignore all contents in lower layers...
rm -rf "${BASE}/${DN}" 2>/dev/null #wipe anything in save layer. 110212 delete entire dir.
mkdir -p "${BASE}/${DN}" #jemimah: files sometimes mysteriously reappear if you don't delete and recreate the directory, aufs bug? 111229 rerwin: need -p, may have to create parent dir.
#also need to save the whiteout file to block all lower layers (may be readonly)...
touch "${BASE}/${DN}/.wh.__dir_opaque" 2>/dev/null
rm -f "$SNAP/$DN/.wh.__dir_opaque" #should force aufs layer "reval".
continue
fi
#110212 recent aufs: .wh.__dir_opaque name changed to .wh..wh..opq ...
if [ "$BN" = ".wh..opq" ] ; then
rm -rf "${BASE}/${DN}" 2>/dev/null #wipe anything in save layer.
mkdir -p "${BASE}/${DN}" #jemimah: files sometimes mysteriously reappear if you don't delete and recreate the directory, aufs bug? 111229 rerwin: need -p, may have to create parent dir.
#also need to save the whiteout file to block all lower layers (may be readonly)...
touch "${BASE}/${DN}/.wh..wh..opq" 2>/dev/null
rm -f "$SNAP/$DN/.wh..wh..opq" #should force aufs layer "reval".
continue
fi
#comes in here with the '.wh.' prefix stripped off, leaving actual filename...
rm -rf "$BASE/$N"
#if file exists on a lower layer, have to save the whiteout file...
#110206 Dougal: speedup and refine the search...
for P in $SFSPoints
do
if [ -e "$P/$N" ] ; then
[ ! -d "${BASE}/${DN}" ] && mkdir -p "${BASE}/${DN}"
touch "${BASE}/${DN}/.wh.${BN}"
break
fi
done #110206 End Dougal.
rm -f "$SNAP/$DN/.wh.$BN" #remove whiteout file. should force aufs layer "reval".
done

#Directories... v409 remove '^var'. w003 remove aufs .wh. dirs.
#w003 /dev/.udev also needs to be screened out... 100820 added var/tmp #110222 shinobar: remove all /dev
#find . -mount -type d | tail +2 | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^root/tmp|^\.wh\.|/\.wh\.|^dev/|^var/tmp' |
#110224 BK revert, leave save of /dev in for now, just take out some subdirs... 110503 added dev/snd
find . -mount -type d | tail +2 | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^root/tmp|^\.wh\.|/\.wh\.|^dev/\.|^dev/fd|^dev/pts|^dev/shm|^dev/snd|^var/tmp' |
while read N
do
#v4.01 graceful exit if shutdown X (see /usr/X11R7/bin/restartwm,wmreboot,wmpoweroff)...
[ "$XRUNNING" = "yes" ] && [ -f /tmp/wmexitmode.txt ] && exit
mkdir -p "$BASE/$N"
#i think nathan advised this, to handle non-root user:
chmod "$BASE/$N" --reference="$N"
OWNER="`stat --format=%U "$N"`"
chown $OWNER "$BASE/$N"
GRP="`stat --format=%G "$N"`"
chgrp $GRP "$BASE/$N"
touch "$BASE/$N" --reference="$N"
done
#100222 a quick hack: BOOTCONFIG written to in init, before timezone set, can cause trouble...
touch /etc/rc.d/BOOTCONFIG
FREEBASEK=`df -k | grep -w "$BASE"| head -n 1 | tr -s ' ' | cut -f 4 -d ' '` #110212 Jemimah #110222 shinobar
rm -f /tmp/snapmergepuppy-nospace #110222 shinobar
#Copy Files... v409 remove '^var'. w003 screen out some /dev files. 100222 shinobar: more exclusions. 100422 added ^root/ftpd. 100429 modify 'trash' exclusion. 100820 added var/tmp #110222 shinobar: remove all /dev
#find . -mount -not \( -regex '.*/\.wh\.[^/]*' -type f \) -not -type d | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^pup_|^zdrv_|^root/tmp|_zdrv_|^dev/|^\.wh\.|^var/run|^root/ftpd|^var/tmp' | grep -v -E -i '\.thumbnails|\.trash|trash/|\.part$' |
#110224 BK: revert, leave save of /dev in for now... 120103 rerwin: add .XLOADED
find . -mount -not \( -regex '.*/\.wh\.[^/]*' -type f \) -not -type d | sed -e 's/\.\///' | grep -v -E '^mnt|^initrd|^proc|^sys|^tmp|^pup_|^zdrv_|^root/tmp|_zdrv_|^dev/\.|^dev/fd|^dev/pts|^dev/shm|^\.wh\.|^var/run|^root/ftpd|^var/tmp|\.XLOADED$' | grep -v -E -i '\.thumbnails|\.trash|trash/|\.part$' |
while read N
do

...


BN="`basename "$N"`" #111229 rerwin: bugfix for jemimah code (110212).
DN="`dirname "$N"`" #111229 "
[ -e "$BASE/$DN/.wh.${BN}" ] && rm "$BASE/$DN/.wh.${BN}" #110212 jemimah bugfix - I/O errors if you don't do this

My head hurts just looking at it.

(these references relevant to Slacko 5.6)
.

[greengeek]

I don't know if this is relevant - but I just repeated the tests I did a while ago trying to create a test pet to understand how to handle .wh files.

(using Slacko 5.6)
First (just for the sake of a learning example) i find the file /root/Web-Server/hiawatha.htm and delete it. This creates a file called /initrd/pup_rw/root/Web-Server/.wh.hiawatha.htm

Look back at the hiawatha.htm file in /root and see that it is now gone. Get rid of the .wh file (or rename it as something else) and hiawatha.htm reappears

Then I make a pet in /root by creating a "whiteout_testpet" directory including the directory /initrd/pup_rw/root/Web-Server/ and in that directory I try to copy the file /initrd/pup_rw/root/Web-Server/.wh.hiawatha.htm from the running filesystem - however the system will not let me do it (see pic 1 below)

No matter - I decide to create the wh file manually (in /root/whiteout_testpet/initrd/pup_rw/root/Web-Server/ - however it will not let me create it manually either!
(see pic 2 below)

That's as far as I ever got.

[nic007]

There must be an init script that deals with whiteout files at bootup and ONLY when included in a savefile/folder (the very top of the layered system). Copying and including a specific white out entry to an adrv for instance, does not work.

[musher0]

Hi guys.

As rufwoof mentioned in his February post above, this is automatically handled by
unionfs / aufs.

Use a script to go counter that OS behavior and you'll mess everything up.
(And need aspirin, too!)

BFN.

[nic007]

Hi guys.

As rufwoof mentioned in his February post above, this is automatically handled by
unionfs / aufs.

Use a script to go counter that OS behavior and you'll mess everything up.
(And need aspirin, too!)

BFN.

But this does not explain why it does not work for an adrv as mentioned because an adrv is booted/mounted to a layer on top of (preferred to) the base sfs like a savefile.

[greengeek]

The save file must contain whiteout files - is that correct? ie: there is no other way to "delete" a file from the base sfs other than for the savefile to contain a whiteout file blanking that file.

So - the save file must receive a copy of the pup-rw whiteout file.

I could not manually copy the whiteout files - so how did BK manually copy them into the savefile??

Can't be a function of the layering system surely - as the layering system knows nothing of what a savefile (or savefolder) actually is. Does it?

@nic007 - did you succeed in getting the whiteout files included in the adrv? Are they actually there??

[greengeek]

What is the ACTUAL difference between a save file and an adrv? A save file contains a "file system" doesn't it? But an sfs does not i think.

Is it possible to build an sfs (xdrv) that also contains a filesystem like a savefile?

Is the layering system treating them differently because it is capable of recognising filesystems as a complete unit.

What does a savefile look like inside?

[mikeb]

hmm i used touch to recreate whiteouts for my save sfs though the contents are then copied back into the ram tmpfs layer at boot. Cannot recall if whiteouts worked in the sfs itself. Perhaps they only work on the topmost layer eg pup_rw. Indeed maybe it needs to be a read/write layer...

Its been a while lol

mike

[rufwoof]

I believe the kernel code only looks for .whiteout files in the upper read/write copyonwrite (cow/save) layer and when found it hides the associated file/folder. adrv is a read only mounted filesystem and as such can't be the upper read/write 'save' area so any .wh files within that (or the main sfs) are considered as being standard files. To get the adrv .wh files to work as intended they'd have to be copied out of adrv to the topmost save/read-write layer manually.

[rufwoof]

What is the ACTUAL difference between a save file and an adrv? A save file contains a "file system" doesn't it? But an sfs does not i think.

Is it possible to build an sfs (xdrv) that also contains a filesystem like a savefile?

Is the layering system treating them differently because it is capable of recognising filesystems as a complete unit.

What does a savefile look like inside?

A sfs is a squashed filesystem. i.e. compressed. The kernel when designed/configured to do so, can mount/use sfs as per any other filesystem, but being compressed its read only.

Yes a sfs can contain another filesystem, but again being within a compressed filesystem that would be mounted a read-only filesystem.

Visualise it as any other compressed file. With gzip for instance you can compress multiple files into a single file that takes up less space, once compressed however you can't change a file contained within that single compressed file without first decompressing and then applying the change and compressing again (recreating the compressed file). For the typical sfs that full extract/change/recompress task is a relatively slow/intensive process.

[mikeb]

See rufwoof for the elegant version of my post

Onions have layers...
ps iirc I used find and touch to create the whiteout copies.

mike

[rufwoof]

See rufwoof for the elegant version of my post

Posted seconds apart. Would have said great minds think alike but this dumb-ass didn't want to insult you Mike