Partially automated remastering of sfs files

How to do things, solutions, recipes, tutorials
Message
Author
artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

Partially automated remastering of sfs files

#1 Post by artsown »

I used manual remastering info from here:
http://www.smokey01.com/coolpup/remastering.html
to create machine specific remasters of frugal installs, including
personal Save files that include installs of the latest Firefox
browser, my bookmarks, Flash player, Sylpheed, Pan, wireless
setup, and the Freecell game. Included also is the MVPS hosts file
in /etc which makes browsing and internet searches much more
sane (ad free). The following script includes merging of the entire
personal Save file. Typically, the new sfs is about 60 meg
larger in size. I have no interest in trying to remove anything.
Sda2 is a 10 gig ext2 partition I use for strictly for this purpose:
-------------- Save this text as makesfs.sh ----------------------------
echo "Please wait. Copying files from original squash file ..."
mkdir -p /mnt/sda2/work
cp -a /initrd/pup_ro2/* /mnt/sda2/work
echo "Merging files from personal Save file ..."
cp -a /initrd/pup_rw/* /mnt/sda2/work
echo "Creating remaster ..."
mkdir -p /mnt/sda2/remaster
cd /mnt/sda2
mksquashfs work remaster/puppy.sfs -noappend
echo "Finished! Remastered file puppy.sfs is in /mnt/sda2/remaster"
---------------------------------------------------------------------------------------
Make sure sda2 is mounted, and then open a terminal. If you locate
the file in /root then type:
sh /root/makesfs.sh

When it finishes, rename puppy.sfs to the particular
name of the puppy .sfs file. I rename the old .sfs and
.2fs files before copying the remastered .sfs file to
their directory. If all is well, I then delete the old
renamed files.

I find I create new .sfs files in four to six minutes using this
script, which is fast enough that I don't mind using it when
a new version of Firefox is available. I manually delete the
/work and /remaster directories when finished with them
so I'm ready to run the script again in the future.

01micko's article on remastering (above link) includes a long line
of text used to create a new iso file that contains the new .sfs
file. I've used that (modified) line in a script I named makeiso.sh
which I run in a terminal. The creation of a new .iso (if that's
your goal) just takes another few seconds. However, I have
little use for remastered .iso files. My interest is strictly in
protecting the work I put into personalizing a frugal installed
pup.

Art

User avatar
Moose On The Loose
Posts: 965
Joined: Thu 24 Feb 2011, 14:54

Re: Partially automated remastering of sfs files

#2 Post by Moose On The Loose »

artsown wrote:I used manual remastering info from here:
http://www.smokey01.com/coolpup/remastering.html
Just a quick thought from my long list of things for later:

If only that which I've changed goes into the main SFS and the old SFS gets turned into an "addon" SFS that gets loaded, under the main one, the new SFS will be small and fairly easy to create. The tick would be automating the action.

User avatar
RSH
Posts: 2397
Joined: Mon 05 Sep 2011, 14:21
Location: Germany

Re: Partially automated remastering of sfs files

#3 Post by RSH »

Moose On The Loose wrote:Just a quick thought from my long list of things for later:

If only that which I've changed goes into the main SFS and the old SFS gets turned into an "addon" SFS that gets loaded, under the main one, the new SFS will be small and fairly easy to create. The tick would be automating the action.
That should be easily to be created (since I'm already using something similar in my Personal Data & Config SFS Manager)!

1. do a snapshot right after X is running of /initrd/pup_rw
- - means: write (echo) everything that is found in there into a first list
2. do your changes and installations etc.pp.
3. do a gain a snapshot of /initrd/pup_rw
- - means: write (echo) everything that is found in there into a second list
4. read out the second list and copy everything that's not found in first list into a directory where to create the new SFS from
5. rename the Puppy's main SFS
6. rename the new created SFS to the main SFS

One could define also lists of directories and/or files that included or even excluded in each case of a remaster!

The real problem here will be then the init script in initrd.gz!

Smarter solution would be to not to rename the SFS Modules and to use the script by mikeb, to mount SFS Modules on top of the top layer. You can find a link to this script at the second post of A vote for a modular use of Puppy Linux.

Another solution could be just to load the new created SFS before X is running, copying all files into the OS and immediately unload the SFS again.

My Personal Data & Config SFS Manager does it exactly this way for a config SFS and a data SFS separately. I'm adding new files or removing included files just by right-click action. Everything und /mnt is excluded and blocked.

My funny little private quick-remaster!

RSH
[b][url=http://lazy-puppy.weebly.com]LazY Puppy[/url][/b]
[b][url=http://rshs-dna.weebly.com]RSH's DNA[/url][/b]
[url=http://murga-linux.com/puppy/viewtopic.php?t=91422][b]SARA B.[/b][/url]

User avatar
Smithy
Posts: 1151
Joined: Mon 12 Dec 2011, 11:17

#4 Post by Smithy »

Code: Select all

-------------- Save this text as makesfs.sh ----------------------------
echo "Please wait. Copying files from original squash file ..."
mkdir -p /mnt/sda2/work
cp -a /initrd/pup_ro2/* /mnt/sda2/work
echo "Merging files from personal Save file ..."
cp -a /initrd/pup_rw/* /mnt/sda2/work
echo "Creating remaster ..."
mkdir -p /mnt/sda2/remaster
cd /mnt/sda2
mksquashfs work remaster/puppy.sfs -noappend
echo "Finished! Remastered file puppy.sfs is in /mnt/sda2/remaster" 
Groovy, can this Quick Remaster be done in ram, no save file just tweaks?
After doing tweaks to a Live Ram Puppy, could this be used to create a new sfs and just dumped to root? Then you can put your newly created sfs on a stick with the other bits.
That would be Quick Quick Remaster :wink:

User avatar
RSH
Posts: 2397
Joined: Mon 05 Sep 2011, 14:21
Location: Germany

#5 Post by RSH »

This method will of course include some files that I've seen often recommended NOT to include in a remaster, like:

- /etc/.XLOADED
- .wh. files
- generally files in /root, /etc and /tmp

Might be suitable for private use only - and probably only for the computer used to remaster the OS.
[b][url=http://lazy-puppy.weebly.com]LazY Puppy[/url][/b]
[b][url=http://rshs-dna.weebly.com]RSH's DNA[/url][/b]
[url=http://murga-linux.com/puppy/viewtopic.php?t=91422][b]SARA B.[/b][/url]

User avatar
Ted Dog
Posts: 3965
Joined: Wed 14 Sep 2005, 02:35
Location: Heart of Texas

#6 Post by Ted Dog »

Ive posted my multisession sloppy remaster. It removes most known duplicates from initrd and almost all tmp files not in temp. the X file must be removed. But it snapshots a running copy so that is different then this I think. It will also show how to get the name of what its running in, Progress popup etc. Feel free to borrow and update..

artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#7 Post by artsown »

Smithy, I just ran across this:
http://murga-linux.com/puppy/viewtopic. ... 88&t=89173
I haven't tried it yet but since the new squash file creation takes up most
of the remastering time, (not the file copying) it looks interesting.

Art

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#8 Post by mikeb »

Check out mksquashfs options... you can choose compression options ... for my save I choose no compression...gives similar speed to tar that way. Without lzma there is a considerable speedup I notice (slax module building for example) Xz would be similar as xz is just a later form of lzma.

Agreed the whiteouts could do with better handling but don't use snap merge as a reference on that.

mike

User avatar
Moose On The Loose
Posts: 965
Joined: Thu 24 Feb 2011, 14:54

#9 Post by Moose On The Loose »

mikeb wrote:Check out mksquashfs options... you can choose compression options ... for my save I choose no compression...gives similar speed to tar that way. Without lzma there is a considerable speedup I notice (slax module building for example) Xz would be similar as xz is just a later form of lzma.

Agreed the whiteouts could do with better handling but don't use snap merge as a reference on that.

mike
Perhaps a tool that took the SFS and 2SF files as inputs instead of doing it on a live system would be better. You could do two frugal installs on the same machine. One can serve as the backup in case you mung the other. Running in the backup one, a tool could let you move things in and out of the SFSs from the 2SF of the normal system.

artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#10 Post by artsown »

Ok, I tried the mksquashfs from here:
http://murga-linux.com/puppy/viewtopic. ... 88&t=89173
and it is indeed fast. Instead of 4 to 6 minutes to remaster, the time is more like
100 seconds ... about 50 seconds to copy files and 50 seconds to make the new squash
file. Thiis is on a twelve or thirteen year old desktop having a 2 ghz p4 and 768 meg RAM.

I used file finder to locate the existing mksquashfs files and found there were copies
in both /usr/bin and /usr/sbin I used the existing symlinks to point to the new
executeable which I renamed. (The old ones had names like mkquashfs4).

This added step of replacing the mksquashfs file is now part of my customization
plan.

I have remastered a number of different puppy versions, and I like to run them without
creating/using a personal Save file. The only artifact that often exists is a message
during bootup that a power fail was detected. I simply choose Ignore, but for the fun
of it, I might next take a look at removing this message.

Art

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#11 Post by mikeb »

Check out mksquashfs options... you can choose compression options ... for my save I choose no compression...gives similar speed to tar that way. Without lzma there is a considerable speedup I notice (slax module building for example) Xz would be similar as xz is just a later form of lzma.

How do you think I disable compression...by hanging my legs out of the window?

have fun

mike

User avatar
Smithy
Posts: 1151
Joined: Mon 12 Dec 2011, 11:17

#12 Post by Smithy »

Artsowen, that power failure message at the boot could be related to Xloaded, I remember a tip to rem out something or delete xloaded.

Mike, you touch type with your toes then :)

Anyone still using pizzagood's edit sfs. Really good, and Don I think added an option to do two compression types at the end. Some of those printer drivers break it (option at beginning to decompress to partition or ram, just defaults to ram).

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#13 Post by mikeb »

Mike, you touch type with your toes then Smile

ever since I learned to juggle.

I forget that we have the option to colour our posts...these technical forums are just too plain....

miswak



artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#14 Post by artsown »

No need to shout, mikeb. I heard you the first time :) I certainly hope you don't start
a fad here of kiddie cartoon printing. The world is far too looney as it is.

Art

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#15 Post by mikeb »

Just want to bring some colour to thee world.

Anyway, ever heard of anyone shouting in green?

As for the loony world we went out in it this afternoon and got reminded of why we avoid it :D

Perhaps I want a new career painting forum threads....for those who have an aversion to white.

Laugh, and the world run a mile,

mike

ps nice ideas by the way in yer thread

User avatar
RSH
Posts: 2397
Joined: Mon 05 Sep 2011, 14:21
Location: Germany

#16 Post by RSH »

No need to shout, mikeb.
Just want to bring some colour to thee world.

Anyway, ever heard of anyone shouting in green?
As far as I know, I never heard of calling it "a shout" if one is using large fonts.

Only if using strictly capital characters (ABCDEFG).

How ever: did never understand that. To me its just another "stupid law" of internet behavior.
[b][url=http://lazy-puppy.weebly.com]LazY Puppy[/url][/b]
[b][url=http://rshs-dna.weebly.com]RSH's DNA[/url][/b]
[url=http://murga-linux.com/puppy/viewtopic.php?t=91422][b]SARA B.[/b][/url]

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#17 Post by mikeb »

Its ok...I saw green as apposed to red....

the therapy is kicking in and I feel fine now :)

mike

lets have a party instead

Image

artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#18 Post by artsown »

Smithy, I located a discussion on removing the .XLOADED file and the fix works to get rid
of the bootup mesage concerning a power fail when running without a personal Save file
after remastering. You add the line:
rm /etc/.XLOADED
to /etc/rc.d/rc.local

To make this stick I created a personal Save file temporarily instead of editing during the
remastering process. That way, I can simply use the full normal remaster script without
interruption and editing.

So this is the second addition to my original procedure (which assumes the use of a personal
Save file initially while doing modifications to puppy). I haven't yet experimented with simply
turning off compression completely in mkquashfs instead of using the version with very
light compression. Been busy with other things here.

Thanks to all for your inputs. Hope my inputs help those interested in remastering machine-
specific and personalised pups we can use without personal Save files.

Art

User avatar
mikeb
Posts: 11297
Joined: Thu 23 Nov 2006, 13:56

#19 Post by mikeb »

Size and speed without any compression are similar to using tar. a 50MB save on this pentium 3 takes half a second to a second to hard drive for example.

I think the xloaded detection is in /etc/profile...its pretty pointless so you might want to remove it...if its not there its in xwin... on windows and been a while and dodgy memory.

mike

artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#20 Post by artsown »

Mike, I tried using no compression and the total remaster time dropped from 100 to 80
seconds while the resulting squash file size increased from 221 meg to 395 meg. The
221 meg was with the minimal compression ... with full compression the file size is
about 200 meg. There is some boot speedup without compression since no decompression
is required ....a drop from about 40 seconds to 30 seconds full to no compression.

I'm working on some other things, and don't know if I'll be running more tests and accumulating
more data or not. Seems to me that users would be very interested in seeing the results
of such tests.

Anyway, thanks for nudging me into trying out the no compression option.

Art

Post Reply