I have 1187 .wh files. Why?

[Dougal]

In the first script (2.14) I did it right, dots escaped. The mistake was only in the second version, for Puppy 2.16. I don't know why, I goofed... Unfortunately, the script is already in 2.16 final. I am sorry.

Ha! I trained myself to use "grep -F" by default... no need to escape anything .

[PaulBx1]

Yeah andrei, the whiteouts go in pup_rw which eventually ends up in the pup_save. Getnikar is right (at least I think so).

I uninstalled an older puppybackup (that came with Puppy 216, or maybe one carried over from Puppy 214 that I had installed earlier, it's hard to tell) and installed pbackup 2.4.0. Strangely, there are some whiteouts for pbackup, which should not need them as it clearly went into my pup_save, not the CD! It should simply have deleted the files when it was done, not created whiteouts. On the other hand, I do need the whiteouts for puppybackup because I want it (which is still on the CD) to be invisible, since pbackup is to replace it. So, now I can't just delete all my whiteouts any more. Here is the list:

Code: Select all

sh-3.00# find /initrd/pup_rw | grep "/\.wh\."
/initrd/pup_rw/etc/.wh.mtab~3441
/initrd/pup_rw/etc/.wh.mtab~
/initrd/pup_rw/root/.packages/moved2cd/.wh.__dir_opaque
/initrd/pup_rw/root/.packages/.wh.puppybackup.files
/initrd/pup_rw/root/.packages/.wh.puppybackup.keyword
/initrd/pup_rw/root/.packages/.wh.pbackup-2.4.0
/initrd/pup_rw/root/.packages/.wh.pbackup-2.4.0.tar.gz
/initrd/pup_rw/root/.wh..events.tmp
/initrd/pup_rw/root/.mozilla/default/Cache/.wh.__dir_opaque
/initrd/pup_rw/root/.mozilla/default/Cache/.wh.038D8B4Ad01
/initrd/pup_rw/root/.mozilla/default/.wh.Cache.Trash
/initrd/pup_rw/root/.thumbnails/normal/.wh.__dir_opaque
/initrd/pup_rw/root/.geany/.wh.geany_socket
/initrd/pup_rw/root/.gxine/.wh.config~
/initrd/pup_rw/root/.pbackup/.wh.__dir_opaque
/initrd/pup_rw/root/.pbackup/locals/.wh.__dir_opaque
/initrd/pup_rw/root/.pbackup/profiles/.wh.__dir_opaque
/initrd/pup_rw/tmp/.wh.wpa_ctrl_2583-0
/initrd/pup_rw/tmp/.wh.wpa_ctrl_2597-0
/initrd/pup_rw/tmp/.X11-unix/.wh.__dir_opaque
/initrd/pup_rw/tmp/.wh..tX0-lock
/initrd/pup_rw/tmp/.wh.tclBVPOX9
/initrd/pup_rw/tmp/.wh.tclNta0Wa
/initrd/pup_rw/tmp/.wh.geany_1180806274.3425
/initrd/pup_rw/usr/local/bin/.wh.puppybackup
/initrd/pup_rw/usr/local/.wh.puppybackup
/initrd/pup_rw/usr/share/doc/pbackup/.wh.__dir_opaque
/initrd/pup_rw/usr/share/applications/.wh.puppybackup.desktop
/initrd/pup_rw/usr/share/icons/.wh.puppybackup.xpm
/initrd/pup_rw/var/run/wpa_supplicant/.wh.__dir_opaque
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2268-0
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2268-1
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2268-2
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2267-0
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2267-1
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2267-2
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2276-0
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2276-1
/initrd/pup_rw/var/lib/pcmcia/.wh.cm-2276-2
/initrd/pup_rw/var/lib/xkb/.wh.server-0.xkm
/initrd/pup_rw/.wh.pbackup-2.4.0.pet.specs

Other whiteouts, I don't know why they are there, like the pcmcia things.

I'm guessing maybe the unneeded whiteouts (for things in pup_rw) are a "feature" of unionfs, which will go away if I switch to aufs?

[andrei]

This is the answer to Dougal (see below) and PaulBx1

PaulBx1, do you run Puppy with pup_save on the harddrive?

This might explain my confusions. I always run it from USB flash drive, with pup_save on USB. I don't know how it works when pup_save is on the hard drive, perhaps in that case pup_save is directly mounted on pup_rw. (When running from USB, pup_save is on pup_ro1, and pup_rw is totally in RAM, periodically saved to pup_ro1; this is done to avoid frequent writes on the flash drive)

In any case, I understand your question, for example this one:
/initrd/pup_rw/root/.packages/.wh.pbackup-2.4.0.tar.gz
It is not clear at all what this whiteout is doing. Very strange. Looks like an aufs bug?

Dougal, your remark is very interesting. Can you please elaborate on this?
Perhaps you could give a scenario, an example, when not having deleted something causes conflict in libraries and segmentation fault? I would think naively, that new libraries will be written over old ones. Remember, the writable layer is always atop the squashes. If you have a library file in squash, and then you write the new file with the same name in pup_save, then the system will see only the new file... How could this lead to an error?

[Dougal]

Remember, the writable layer is always atop the squashes. If you have a library file in squash, and then you write the new file with the same name in pup_save, then the system will see only the new file... How could this lead to an error?

Well, there are various (strange) scenarios... I have seen strange things happen in Puppy.

The main thing to remember is that it's very likely that Puppy users will install "foreign" packages -- from Slackware for example.

I think people had some problem with Geany and libvte, I can't remember exactly what it was, but that brings the problem of apps (like Geany) with "soft" dependencies -- it searches for vte when you start it, but can run without it.
Maybe some app will search for an app that's in Puppy, but Puppy has an older version, or compiled with different options?

Also, foreign libraries/apps are likely to be compiled for /use/local, while in Puppy nearly everything goes in /usr, so you might not be able to over-write the old libraries with the new ones (at least not when it comes to sub-directories), which is especially problematic with applications: if you can't delete the one in /usr, it'll probably be found first when you try running the app (unless you run it with a ful path).

Again, this is all theoretical (though I'd say that theoretically being able to delete files is important!), but I've seen cases where deleting something did solve a problem...

[PaulBx1]

PaulBx1, do you run Puppy with pup_save on the harddrive?

Yes. That explains the confusion. Puppy is so flexible it is hard to keep these discussions straight!

[PaulBx1]

Dougal, you wrote this earlier:

I put a little loop into rc.shutdown, that runs every 20 boots and finds all whiteout files and deletes the unneeded ones (i.e. the ones created when you delete a file that you created in the first place).

Could you post this code here, and also where in rc.shutdown you put it (if it's critical in that respect)? I might do the same since my whiteouts keep multiplying, and I'd like to fool with this code anyway without re-inventing the wheel (my script writing skills have atrophied quite a bit).

[Dougal]

Here you go. This runs the cleaning every 20 boots, but you can change it to as frequent as you like.

Code: Select all

## Dougal: add cleaning out of "orphaned" whiteout files:
# where pup_save is mounted:
OLDFILESMNTPT="`mount | fgrep '/initrd/pup_' | fgrep -v 'none on /'| grep 'ext[23]' | tr '\t' ' ' | tr -s ' ' |cut -f3 -d' '`"
if [ "$OLDFILESMNTPT" ] && [ $PUPMODE -ne 77 -a $PUPMODE -ne 2 ]; then
  echo -n "0" >>/etc/.bootcount
  COUNT="`cat /etc/.bootcount`"
  # the "20" below is how many boots between cleanups
  if [ ${#COUNT} -ge 20 ] ; then
    echo -n "" >/etc/.bootcount
    echo -n "Cleaning out whiteout files..."
    # all mountpoints of various sfs files (and pup_save)
    ALLPOINTS="`mount | grep -F '/initrd/pup_' | grep -F -v 'none on /' | tr '\t' ' ' | tr -s ' '| cut -f3 -d' ' | tr '\n' ' '`"
    NUM=0
    cd $OLDFILESMNTPT
  find . -mount -type f -name ".wh.*" ! -name "*__dir_opaque*" | grep -o '/.*' |
    while read ONEOPAQUE
    do
	  DIR=${ONEOPAQUE%%.wh.*}
	  WHNAME=${ONEOPAQUE##/*/}
	  [ "$WHNAME" = "$ONEOPAQUE" ] && WHNAME=${ONEOPAQUE#/}
	  FILENAME=${WHNAME#.wh.}
	  for APOINT in $ALLPOINTS
	  do
	    ## continue 2 -- go to next ONEOPAQUE!
	    [ -e "${APOINT}${DIR}$FILENAME" ] && continue 2
	  done
	  ## never get to this if file is in one of the points above
	  rm -rf ${OLDFILESMNTPT}"$ONEOPAQUE"
	  let NUM=NUM+1
    done
    echo "$NUM files removed"
  fi #if [ ${#COUNT} -ge 20 ]
fi
##

I added it towards the end of the script, before the "stray partitions" part.

[craftybytes]

Just to jump in on this discussion -

Yesterday I had a problem with running my Opera web browser in Puppy - it seems to have slowed down a fair bit - so I went in and deleted all the cache, cookie & search & history files within opera itself - speeded it up but not as much as I thought it would.

A little background - I boot puppy 2.15CE from live-CD and 'save' to hard drive "pup_save" file(s) - there are 5 at the moment..

Anyway after finishing the web surfing session - I then redid the 'clearing' of the relevant files in opera again - checked in ROX - no hidden files left (no .wh.xx files either) - so felt that all was cleaned out. - normally ROX does show any hidden '.wh.xx' files as well so all was cleaned...

As I had to do something in my other distro (MEPIS 3.4-3) - i also decided to check the latest puppy 'pup_save' file that I had just finished with before booting into MEPIS - I mounted it using the "loop" option..

What I found amazed me - I thought I had cleaned out all the ".wh.xx" files from the 'pup_save' file - but lo and behold - the opera cache still showed a whopping 30384 .wh files - all zero byte size - but still as a file name so this MUST take up some space in the storage medium - how much I'm not certain..

How come the 'cleaning' in Puppy did not actually do as I thought it should - ROX showed that all 'hidden' files had been removed (including all the .wh.xx files) - so what is going on ..?

This time whilst in MEPIS - I used the 'wipe' command in a terminal screen to delete all these pesky '.wh.xx' files from the loop mounted 'pup_save' file..

Maybe this needs to be clarified..?

crafty.
.

[PaulBx1]

Hmmm, if you cleaned them out maybe we are not going to be able to answer any questions. Got a backup of that pup_save to play with?

I'd be wondering if the Puppy "find" command can see them, like those scripts we mentioned above. Maybe Rox has some problem. Of course you have to left click on the eyeball thingy to see those hidden files, I assume you know that...

Those 30k files may take no room per se, but they'd have to be clogging things where a program (including unionfs itself) has to dig through them all to do its job. Just a guess...

Thanks Dougal, I'll take a look...

[PaulBx1]

Funny, when I ran this stand-alone it went and deleted a pile of whiteouts (I added an echo for each deletion). But at the end it said "0 files removed". I don't see why, but it's not very important...

I was looking at the code. For me the variable ALLPOINTS is equal to "/initrd/pup_ro2 /initrd/pup_rw /". So, what it looks like it is doing is wiping out each whiteout that does not correspond to a real file in one of those three points. All well and good (although I wonder about "/"...)

However I still have the question; why would there ever be a whiteout that masks a file in pup_rw? If there ever is such a thing, this script will not delete it. What I think should happen in that case, is that both the whiteout and the file itself should be deleted (since whiteouts are just a kludge that has to be used where files cannot simply be deleted, as in read-only media - but they always can be if they reside in /initrd/pup_rw).

Another question. Whiteouts, if I understand them properly, only blank files that are lower in the hierarchy than the place the whiteout exists. Does this script take that into account? My guess is no, and that if there is a file higher in the hierarchy but not lower, then this script will leave the whiteout alone (which I guess is pretty harmless unless there are a pile of such cases).

[Dougal]

why would there ever be a whiteout that masks a file in pup_rw?

I think it might be a bug in unionfs.
That's what that loop is about: getting rid of "orphaned" whiteouts.

Try this: open a ternimal and type:

Code: Select all

touch somefunnyname

That will create the (empty) file ~/somefunnyname
Now delete it

Code: Select all

rm -f somefunnyname

The file will be gone, but there will be a whiteout for it: ~/.wh.somefunnyname

So every time you add a new file to the filesystem and then remove it, a new "orphan" whiteout is created...
That's probably what happens with the cache -- it uses temp names, so the whiteouts just keep accumulating.

Oh, the reason NUM isn't incremented is that the loop is run as a sub-shell (since the data is piped to it), so the modified parameter isn't recognised in the parent process... I should change it, I guess.