Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 11 Dec 2019, 13:58
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
merge sfs and save files into a directory and apply whiteout
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [11 Posts]  
Author Message
s243a

Joined: 02 Sep 2014
Posts: 2204

PostPosted: Tue 12 Mar 2019, 04:49    Post subject:  merge sfs and save files into a directory and apply whiteout
Subject description: files. Could be used as an external aproach to remastering.
 

I wrote a script that merges sfs files and a save file into a single directory then updates the directory based on the whiteout files in the save folder.

https://pastebin.com/pvSANipi

The script is preliminary and I haven't tested all use cases. I'll update this post later.

This could be used as an offline way of remastering a puppy. Perhaps though it isn't as robust an approach as constructing a layered file system.

Currently this post is just a placeholder. I'll write more later.
Back to top
View user's profile Send private message Visit poster's website 
foxpup


Joined: 29 Jul 2016
Posts: 964
Location: europa near northsea

PostPosted: Tue 12 Mar 2019, 06:40    Post subject:  

Interesting tool s243a.

I looked to the script and did not understand it at all. Embarassed
But I am almost noob scripting.

I thought it would be simple.
This is what I think:
- unsquash all the sfs to the same working directory (wd)
- copy all files from savefile to the same wd
- remove all files with the whiteout prefix (suffix?) and the corresponding files without the whiteout prefix in de wd
- mksquashfs the wd
Is this wrong? How?
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2204

PostPosted: Tue 12 Mar 2019, 12:57    Post subject:  

foxpup wrote:
Interesting tool s243a.

I looked to the script and did not understand it at all. Embarassed
But I am almost noob scripting.

It's partly my fault also because I added some complexity that wasn't strictly required to get the job done.

Quote:

I thought it would be simple.
This is what I think:
- unsquash all the sfs to the same working directory (wd)
- copy all files from savefile to the same wd

This is correct. However, rather than unsquashing the sfs file I mount each sfs file at "/mnt/wksfs". Here is some trimmed down code :

around line #240 (but trimmed down and simplified
Code:

while read -r -u9 file1 ; do
    mount -v -t squashfs $file1 /mnt/wksfs

   mv_or_copy_files /mnt/wksfs $target_dir

   umount /mnt/wksfs
done 9< <( find $sfs_dir -maxdepth 1 -type f -name "*.sfs" | \
              grep -v -e "^$sfs_dir"/'f.*' | \
              grep -v -e "^$sfs_dir"/'z.*' )


Code:

mv_or_copy_files()                                                 #Line number 120
  while IFS=$'\0' read  -r -d $'\0' -u10 fm_filePath_prefixed ; do #Line number 141
    #
    mkdir -p `dirname $to_filePath_prefixed`                       #Line number 161
    cp -a -u  $fm_filePath_prefixed $to_filePath_prefixed          #Files are only copied if they are newer
  done 10< <( find $s_folder_fullpath -type f -name "*" -print0 )  #Line number 165
}



The input to the above loops comes from the find command at the end of the function:
https://blog.famzah.net/2016/10/20/bash-process-null-terminated-results-piped-from-external-commands/

In the first find example we have:
(Not valid code)
Code:

find $sfs_dir -maxdepth 1 -type f -name "*.sfs" \ #Find all files the end in .sfs (search only one level deep.
grep -v -e "^$sfs_dir"/'f.*' #exclude files that start with "f" (These are firware)
grep -v -e "^$sfs_dir"/'z.*' #exclude files that start with "z" (these are kernal modules)


In the second example we're finding all files within a given directory
Code:

find $s_folder_fullpath -type f -name "*" -print0

* the -type f option means find only files (no directors). It also seems to include symlinks although this isn't clear from the manpage.
* the -print0 option means to use the "null character" as the field seperator. This can help with unusually file names, However, I don't think my code is perfectly robust this way.

Both loops are reading and writing from a file descriptor. At the end of the loop we have:
Code:

done 10< <( command )

the file descriptor is "10" or more properly written as "&10" (but not here). The reason to use a file descriptor is so that within the while loop we can accept terminal input wihout interfering with the read at the top of the loop. Terminal input comes from standard input (i.e. &1). I use standard input for debugging purposes. For instance I can pause the script with:
Code:

read -p "Press enter to continue" <&1

and then read from the screen a given set of echo statements that I printed out.

At the top of the loop the file descriptor input is showon with the -u10 option.

Quote:

- remove all files with the whiteout prefix (suffix?) and the corresponding files without the whiteout prefix in de wd

This is correct:
Code:

apply_white_out_files(){ #Line number 171
  case
...
  *) #Line number 207
...
    fname=${wh_fname#.wh.} #https://lwn.net/Articles/324291/
    rm $to_dirPath_prefixed/$fname 
    rm $to_dirPath_prefixed/$wh_fname 
    esac
...
}

except that there is also a type of whitout file that means that the current directory is opaque (in other words it blocks all lower layers from showing up in the directory. In this case we replace the directory with the directory from the save folder.

Code:

  case $wh_fname in
 '.wh..wh..opq') #Around line #190
    rm -rf $to_dirPath_prefixed
    cp $fm_dirPath_prefixed $to_dirPath_prefixed
   rm $to_dirPath_prefixed/$wh_fname       


Finally there are two types of whitout files that I don't know what they do so I just delete them:
Code:

     '.wh..wh.plnk'|'.wh..wh.aufs') #https://unix.stackexchange.com/questions/92287/aufs-whiteout-removal
...
       rm $to_dirPath_prefixed/$wh_fname
       ;;   


Quote:

- mksquashfs the wd
Is this wrong? How?


You could do this if you want to just replace the main puppy sfs file. Or alternatively you could build a new ISO.

Edit:; see:make_ISO_Fm_slitaz-rootfs
https://pastebin.com/5BfvZKXy

Requires:
make-tazpup_functions.sh
https://pastebin.com/K7bgqLFE

Last edited by s243a on Tue 12 Mar 2019, 23:35; edited 2 times in total
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2204

PostPosted: Tue 12 Mar 2019, 15:33    Post subject:  

Some use notes. The actual script starts at line #261. Everything above that defines the functions that will be used.

Code:

#This directory is where the file descriptors will write (don't change).
mkdir -p /tmp/mv_or_copy_files

#Remove file descriptors and mount points. This line probably isn't
#necessary because we are trapping the errors and removing the file
#descriptors
remove_fille_descriptpors

#This statment must occur after the "remove_fille_descriptpors" statment. (will fix spelling error later)
trap remove_fille_descriptpors EXIT SIGKILL SIGTERM

#This is the location where the sfs are stored. Currently files that start
#with z or f are excluded. If you want to change this remove the #statments (around line #254):
#grep -v -e "^$sfs_dir"/'f.*' | \
#grep -v -e "^$sfs_dir"/'z.*'
#
#
sfs_dir=/mnt/sdb1/TazPup/64/a1
#This is the directory where the sfs file will be merged

#Used to define the $target_dir variable. See below.
root=/initrd/mnt/dev_save/slacko64save

#This is the folder where the files will be merged into
target_dir=$root/tazpup-builder/slitaz-rootfs-new

#This is the location of the save folder. If you are using a save file then first mount it (I believe you
# can click on it and select view to do this) and the path of the mounted file should be located in
#the /mnt folder.
savedir=$sfs_dir/tazpupsave
 
#This is the function that merges the save file
merge_sfs_and_save_fils $sfs_dir $target_dir
#This is the function that applies the whit out files and then removes the whiteout files.
apply_white_out_files $savedir $target_dir


Note that if you have a sandbox (e.g. sandbox.sh in fatdog64), I recommend running the script in a sandbox since it contains "rm -rf" statements. I ran the script without a sandbox and had no issues but be warned. Also note that I haven't tested the merged file yet.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2204

PostPosted: Tue 12 Mar 2019, 15:44    Post subject:  

I posted this message but It doesn't seem to be sending:

Quote:

Quote:
Hi s243a,

I am very interested in the topic of your new thread and the image below shows my three relevant files in /root/Lab as:

If you downloaded it from pastbin first convert the line endings to unix style using dos2unix. Alternatively, you can copy the raw text from pastbin into geany and the line endings should be correct. Otherwise, geany has tools to convert the line endings.

Quote:

(1) copy of sfs file (with a and z drv already merged)

If you want the kernal modules merged that's cool. Otherwise you can remove them by deleting the /lib/modules folder once extracted.

Quote:

(2) your downloaded code (executable and named s243a)

(3) copy of my current save file (200 M).

I would like to try your code and then report the result to you by PM. If that is OK with you, any extra detailed instruction you can add would help me.

Cheers
Mike


I added some use notes at:
http://murga-linux.com/puppy/viewtopic.php?t=115640

If you want I can add some dialog menus so that you don't have to edit the code but I'm not sure the time-frame (maybe within a week).
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 3675

PostPosted: Tue 12 Mar 2019, 17:45    Post subject:  

Why not overlay (or aufs) the two (or more) folders into a top view ... and mksquashfs that top view?
_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2204

PostPosted: Tue 12 Mar 2019, 18:52    Post subject:  

rufwoof wrote:
Why not overlay (or aufs) the two (or more) folders into a top view ... and mksquashfs that top view?


I did note above that the approach, which you suggest, might be more robust. However, I'm not sure whether or not if would use any more ram.

One reason I didn't take that approach is I don't understand it yet. Another, reason is that I was thinking of adapting the scipt to move files rather than copy them. The idea is to strip out each package from an OS in order to create the underlying skeleton as part of a build system. I might try this for lucid puppy as an example since it is a pretty lightweight puppy and supports multi-session cd/dvd saves better than more recent versions.

That all said it is on my list to learn more about the approach that you suggest. However, taking an alternative approach let me learn a bit about white-out files Smile
Back to top
View user's profile Send private message Visit poster's website 
rufwoof


Joined: 24 Feb 2014
Posts: 3675

PostPosted: Wed 13 Mar 2019, 06:13    Post subject:  

Quite simple. for example if you have a sfs 'my.sfs' and a changes folder 'changes' then to mount the two with a top layer of 'top' ...
Code:
mkdir changes sfs top
mount -r -t squashfs ./my.sfs sfs # mount my.sfs to sfs folder mountpoint
mount -t aufs -o br=changes:sfs none top # aufs mount

Something like mksquashfs top top.sfs ... then creates a sfs of the merged top folder. You could then close that down
umount aufs
umount sfs
... and copy the newly created top.sfs to my.sfs, clear out the changes folder ... and you've in effect remastered all of the prior changes into a new version of my.sfs

_________________
( ͡° ͜ʖ ͡°) :wq
Fatdog multi-session usb

echo url|sed -e 's/^/(c/' -e 's/$/ hashbang.sh)/'|sh
Back to top
View user's profile Send private message 
foxpup


Joined: 29 Jul 2016
Posts: 964
Location: europa near northsea

PostPosted: Wed 13 Mar 2019, 16:42    Post subject:  

@s243a
@rufwoof

Thank you both for your explications! Very instructive.
I will have to read and try this out.

Btw, s243a, I like the idea of stripping to the 'skeleton'/core and then rebuild with just what you need/want.
I wonder if it could be done already with pkg from sc0ttman?
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 1842
Location: not Bulgaria

PostPosted: Tue 02 Apr 2019, 01:54    Post subject: Re: merge sfs and save files into a directory and apply whiteout
Subject description: files. Could be used as an external aproach to remastering.
 

s243a wrote:
I wrote a script that merges sfs files and a save file into a single directory then updates the directory based on the whiteout files in the save folder.


I like this kind of work, which is based on understanding of what happens in a system using some kind of overlay (in terms of whiteout files etc), but doesn't need using overlay in order to do the merging. Reminds me of tinycorelinux showing how to mount and merge squashed filesystems using symlinks rather than aufs or overlayfs. Of course, using overlay to get merged version of sfs and savefile also works - but nice to illustrate alternative methodologies.

wiak

_________________
Tiny Linux Blog: http://www.tinylinux.info/
Check Firmware: www.murga-linux.com/puppy/viewtopic.php?p=1022797
tinycore/slitaz: http://www.murga-linux.com/puppy/viewtopic.php?p=990130#990130
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2204

PostPosted: Thu 04 Apr 2019, 22:08    Post subject:  

Just as a note on this, it mostly works but I had some weird errors the last time I tried it that may or may not be related to my code. I do know that I need to take more care when creating directories to get the permissions correct. One way to do this is to use the cpio command.

This concept is temporarily on hold because of other things that I"m working.

I apologize for not responding to some PMs but my sent box is full and I haven't gotten around to moving the old messages to a suitable alternative location.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 1 of 1 [11 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0591s ][ Queries: 11 (0.0067s) ][ GZIP on ]