Remastering remastering

A home for all kinds of Puppy related projects
Message
Author
User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

Remastering remastering

#1 Post by pa_mcclamrock »

I'm working on an app tentatively entitled "Remaster Express" to make the Puppy custom live-CD remastering process more versatile and more user-friendly, with a GUI that persists throughout the process, incorporates more options than the existing remaster script (/usr/sbin/remasterpup2), and takes on the selected GTK+ 2.0 theme (by way of Gnocl). Here are links to some screenshots of what I've got so far. [EDIT 2009 Feb 14: Don't look at these any more, look at the completed set of screenshots with the links in the message I just posted below.)

Initial "Welcome" window: http://www.geocities.com/pa_mcclamrock/remaster01.gif

Remaster Slimmer window, for removing built-in packages before remastering: http://www.geocities.com/pa_mcclamrock/remasterslim.gif

Step Two, estimate working space needed: http://www.geocities.com/pa_mcclamrock/remaster02.3.gif

Show estimated working space and select working partition: http://www.geocities.com/pa_mcclamrock/remaster02.6.gif

Step Three, show information about remastering while making heap big squashfile: http://www.geocities.com/pa_mcclamrock/remaster03.gif

Step Four, verify that contents of /tmp/root include everything desired: http://www.geocities.com/pa_mcclamrock/remaster04.gif

Step Five, do the same with /tmp/etc and specify hardware configuration if desired: http://www.geocities.com/pa_mcclamrock/remaster05.gif

Step Six, edit isolinux.cfg if desired: http://www.geocities.com/pa_mcclamrock/remaster06.1.gif

Step Seven, rename pup_xxx.sfs file if desired:
http://www.geocities.com/pa_mcclamrock/remaster07.gif

I don't have the rest done yet, but I do have several questions that I hope somebody here can answer.

* How do you best estimate the size of the pup_xxx.sfs file compared to the uncompressed size of the directories that go into it? The "remasterpup2" script just seems to guess that maybe it will be about 1/3 as big, and maybe you'll need, oh, 150 to 200 MB of additional free space if you're going to save an ISO. Is it possible to do better than this? As I said above, I'd like to have a progressbar to show what percentage of stuff has been squashed into the squashfile and what percentage remains to be squashed, but this would seem to require a pretty good estimate of how big the whole thing will be when done.

* I need to make sure I know what to do with the various "packages" files in /root/.packages. I observe that "packages.txt" has the complete list of official packages from the current Puppy Unleashed; "livepackages.txt" is the same as the "Official Puppy 4" list you see in PETget; "livepackages2.txt" and "livepackages3.txt" are the same as Official Puppy 2 and Official Puppy 3 respectively; and "alienpackages.txt" lists the unofficial packages you have installed. To this I have added "slimpacks-xxx.txt," a modified list of built-in packages the user might wish to uninstall with "Remaster Slimmer" before remastering.

What I want to do, I think, is to (1) copy "packages.txt" intact to the /tmp/root/.packages directory for inclusion in the new CD; (2) copy the various "livepackages" files and "slimpacks-xxx.txt" to /tmp/root/.packages too, but then (3) move all the "on" lines out of the "livepackages" copies and merge them into /tmp/root/.packages/slimpacks-xxx.txt, since the packages named in these lines will become built-in packages in the new CD and will be uninstallable with Remaster Slimmer but not with PETget; and (4) omit "alienpackages," since all the packages from there will become built-in packages in the new CD. Does that sound right?

* Following line 196, "#now copy the files... ," "remasterpup2" omits "boot.cat" from the list of files to be copied from the existing live-CD, and includes something called "goofy.16" instead. What's this, and why?

* Are all these "syncs" really necessary? Almost everywhere you do anything with files in "remasterpup2," it seems, there's a "sync" before and after. Would there be any significant chance of disaster if you didn't have quite so many "syncs"?

* All references to "zdrv" can be omitted for Puppy 4.1 and later, can't they?

Thanks in advance for any help!

David McClamrock
Last edited by pa_mcclamrock on Sat 14 Feb 2009, 11:14, edited 7 times in total.
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#2 Post by Lobster »

The last easy remaster was from Dougal and worked very well.
It was integrated into the main Puppy but was easier in Dougals version, so it might be worth contacting him directly.

I look forward to this and it would be a welcome feature to the next Puppy after Deep Thought or if ready for Deep Thought.

8)
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#3 Post by pa_mcclamrock »

Lobster wrote:I look forward to this and it would be a welcome feature to the next Puppy after Deep Thought or if ready for Deep Thought.
Thanks! I think I'm fairly far along with it, and it may even be ready for 4.2 Alpha4, depending on (1) how soon Alpha4 comes out and (2) how soon I can get specific answers to my questions. :)

David McClamrock
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#4 Post by technosaurus »

I had a look at the remasterpup2 script a while back and used it to make a mkaddonsfs script ... started toward what you are working on - I can upload my scripts tonight when I get home, but the original script is already posted in the forum - I'll edit my post with a link when I find it.... there is also a dir2iso script floating around somewhere - I have it set up to where you manually add the other files (except pup_XXX.sfs) from the cd

Edit: http://www.murga-linux.com/puppy/viewtopic.php?t=37856

& dir2iso http://www.murga-linux.com/puppy/viewto ... 34&t=36573
Last edited by technosaurus on Thu 05 Feb 2009, 16:47, edited 2 times in total.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
tasmod
Posts: 1460
Joined: Thu 04 Dec 2008, 13:53
Location: North Lincolnshire. UK
Contact:

#5 Post by tasmod »

David,
I'm not a programmer or anything but I was just looking at the pwidgets and noticed that Puppy Space seems to reflect the puppy space plus amount used. Is that any help?

Rob Lane

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#6 Post by technosaurus »

I wouldn't remove the zdrv because it can still be used and I do frequently as a way to test how an sfs will work with default versions and puplets (if you name an addon to zdrv you don't need a pupsave file to load it - its automatic)

for /root/.packages/*.files - I left them in to allow people to more easily remove programs from puplets (petbegone only works for default apps). Either way is fine - depends on how you want it to be used I guess

syncs - I think? it is to minimize disk usage... otherwise you need the space of the full expanded file system + the size of your sfs (I got rid of them in my script, but only because I wanted the user to have more control)

Is boot.cat automatically generated when making the iso bootable? & goofy.16 the boot picture (usually tux)?
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#7 Post by pa_mcclamrock »

tasmod wrote:I'm not a programmer or anything but I was just looking at the pwidgets and noticed that Puppy Space seems to reflect the puppy space plus amount used. Is that any help?
Um, don't know offhand, but I figured out a way to avoid having to predict the exact size of the pup_xxx.sfs file. (Never do things the hard way, I say, if you can legitimately get away with doing them the easy way--but thanks for trying to help me do this the hard way!)

(1) For the purpose of estimating the amount of working space needed, I'll just compare the calculated size of the uncompressed filesystems (plus a bit extra for the kernel and other files to be copied from the existing live-CD) with the size of the ISO I get when this program actually works, and then overestimate somewhat to allow for the possibility that other people's systems may include more already compressed files than mine does.

(2) For the purpose of the progressbar, I'll just go by a count of the total number of files to be copied. The progress may be a bit uneven, what with big and little files, but I'm pretty sure nobody will complain too much. There will be a little "program output" window at the bottom, in which the user will be able to see filenames whizzing past at top speed (if the files are small) or not (if they're big) during this step of the process.
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#8 Post by pa_mcclamrock »

technosaurus wrote:I wouldn't remove the zdrv because it can still be used and I do frequently as a way to test how an sfs will work with default versions and puplets (if you name an addon to zdrv you don't need a pupsave file to load it - its automatic)

for /root/.packages/*.files - I left them in to allow people to more easily remove programs from puplets (petbegone only works for default apps). Either way is fine - depends on how you want it to be used I guess

syncs - I think? it is to minimize disk usage... otherwise you need the space of the full expanded file system + the size of your sfs (I got rid of them in my script, but only because I wanted the user to have more control)

Is boot.cat automatically generated when making the iso bootable? & goofy.16 the boot picture (usually tux)?
I looked at the "mkisofs" man page and, sure enough, boot.cat is automatically generated. Don't know what goofy.16 is, but I can't copy it since I don't have it on my live-CD.

Thanks--I think I've got enough answers to proceed--hope I'll get done pretty soon!
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

seeker
Posts: 89
Joined: Thu 27 Nov 2008, 00:43
Location: Indiana, USA

#9 Post by seeker »

Step Two, estimate working space needed:
This is nit picking, but not everyone has the desktop drive icons enabled. Maybe a reference to Pmount is in order?

By the way, I really like what you are doing with gnocl and tcl/tk.

Seeker

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#10 Post by pa_mcclamrock »

seeker wrote:This is nit picking, but not everyone has the desktop drive icons enabled. Maybe a reference to Pmount is in order?
Evidently so. A quick change from "click the desktop icon" to "click the desktop icon or use Pmount" should do the job.
By the way, I really like what you are doing with gnocl and tcl/tk.
Thanks! I think I'm almost ready to try the complete remastering process with Remaster Express. The Gnocl progressbar seems to be easier to get working right than a Tk "ttk::progressbar"; I've done test runs of the "squashfile-making" part of the process with a little squashfile (incorporating the contents of /bin only) instead of a big one, and it works. For making an ISO and burning a CD, the procedures I already use in WISH CD-Writer (http://www.geocities.com/pa_mcclamrock/wishcd.html) should work with little or no modification, and I'm pretty sure I've figured out how to extract the relevant numbers from the "mkisofs" and "cdrecord" output to make the progressbar work. If I give it a try tonight, though, I'll run a serious risk of staying up too late squashing bugs, and I've got to get to work tomorrow morning. :roll: Soon, very soon!

David McClamrock
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#11 Post by Pizzasgood »

goofy.16 is the name of a splash image that isolinux can display during the pause before it starts booting. It's an arbitrary name. Way back in the early 1.0.x days I made a splash screen but couldn't decide on which font was best, so I uploaded several versions named after the fonts. Barry chose the one using the goofy font, and for some reason left it named goofy.16 rather than renaming it to something like boot.16 or image.16. Then he continued using that name in future releases using completely different images.

We haven't used a splash image since we switched to Puppy 2.x.

Until 4.2 that is. Now we have one again. This time I've revised the system somewhat, so that the USB installs can also have it, and can have the pause so they can add things like 'puppy pfix=ram' or whatever.

So you need to know how the new system works:
There is now a directory at /usr/lib/syslinux/files/ which contains a number of files. They should all appear in the ISO. They're also copied onto a USB drive when installed to one. This is done with an *, not by hardcoding filenames.

There is a catch: the syslinux.cfg file in that location is not quite complete, because the final line depends on how Puppy is being installed (or whether being placed on a CD). It's the "append" line that looks like this for a CD version:

Code: Select all

append initrd=initrd.gz pmedia=cd
In addition to appending the append line, the file also needs to be renamed to isolinux.cfg for a CD to use it. (USB installs use it as syslinux.cfg, except when they have an ext2/3 filesystem, in which case they want extlinux.conf - this naming convention is part of the syslinux suit, not my doing...)

This way we don't have hardcoded files, so if somebody wants to add a second screen to the pause (like the F2 screen in 4.2) they don't have to go and update all the remaster scripts. And they are also used in USB installs, because I updated the install script to look in /usr/lib/syslinux/files too.
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#12 Post by pa_mcclamrock »

Here are links to the completed set of screenshots.

Initial "Welcome" window: http://www.geocities.com/pa_mcclamrock/remaster01.1.gif

Remaster Slimmer window, for removing built-in packages before remastering: http://www.geocities.com/pa_mcclamrock/remasterslim.gif

Step Two, estimate working space needed: http://www.geocities.com/pa_mcclamrock/remaster02.5.gif

Show estimated working space and select working partition: http://www.geocities.com/pa_mcclamrock/remaster02.7.gif

Step Three, show information about remastering while making heap big squashfile: http://www.geocities.com/pa_mcclamrock/remaster03.gif

Step Four, verify that contents of /tmp/root include everything desired: http://www.geocities.com/pa_mcclamrock/remaster04.gif

Step Five, do the same with /tmp/etc and specify hardware configuration if desired: http://www.geocities.com/pa_mcclamrock/remaster05.gif

Step Six, edit isolinux.cfg if desired: http://www.geocities.com/pa_mcclamrock/remaster06.1.gif

Step Seven, rename pup_xxx.sfs file if desired: http://www.geocities.com/pa_mcclamrock/remaster07.gif

Step Eight, select "Burn Only," "Save ISO Only," or "Save ISO & Burn": http://www.geocities.com/pa_mcclamrock/remaster08.gif

Step Nine, finish up by displaying more reading matter while live-CD gets burned, or at least ISO gets created: http://www.geocities.com/pa_mcclamrock/remaster09.gif
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#13 Post by pa_mcclamrock »

Pizzasgood wrote:We haven't used a splash image since we switched to Puppy 2.x.

Until 4.2 that is. Now we have one again. This time I've revised the system somewhat, so that the USB installs can also have it, and can have the pause so they can add things like 'puppy pfix=ram' or whatever.

So you need to know how the new system works:
There is now a directory at /usr/lib/syslinux/files/ which contains a number of files [details follow].
Um, always looking for the easy way if feasible, I ask: If I just (1) list all the files I see on a Puppy 4.2 Alpha4 live-CD (except pup_420.sfs and boot.cat) for copying to the new ISO and/or CD, and (2) include copies of syslinux.cfg under the names isolinux.cfg and extlinux.conf, will that do the job for the 4.2 version of Remaster Express?
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#14 Post by Pizzasgood »

The key thing is to hard code as few names as possible, so that we can modify the files included in the ISO later without having to edit half a million scripts to handle the new names.

Now that I think about it, I guess the new method doesn't really affect a remaster so much as it affects installations and such. Unless you're allowing a remaster without an existing CD/ISO, in which case you would need to go into that directory to get the files, and append this line to the new isolinux.cfg file:

Code: Select all

append initrd=initrd.gz pmedia=cd
But when it comes to remastering from an existing CD / ISO, you could get away with just doing something like this:

Code: Select all

mount -o loop old_puppy.iso old_iso_dir
rsync -r --exclude=boot.cat --exclude puppy_[0-9][0-9][0-9].sfs old_iso_dir/ new_iso_dir/
umount old_iso_dir
That would automatically copy everything, directories included, from the old iso into the new one, with the exception that it will exclude pup_xxx.sfs and boot.cat. (I used rsync because cp doesn't have an exclude option).

Then just create the pup_xxx.sfs file.
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#15 Post by pa_mcclamrock »

Pizzasgood wrote:The key thing is to hard code as few names as possible, so that we can modify the files included in the ISO later without having to edit half a million scripts to handle the new names.
Fair enough.
Now that I think about it, I guess the new method doesn't really affect a remaster so much as it affects installations and such. Unless you're allowing a remaster without an existing CD/ISO [ . . . ]
I'm not!
But when it comes to remastering from an existing CD / ISO, you could get away with just doing something like this:

Code: Select all

mount -o loop old_puppy.iso old_iso_dir
rsync -r --exclude=boot.cat --exclude puppy_[0-9][0-9][0-9].sfs old_iso_dir/ new_iso_dir/
umount old_iso_dir
Let me see, in Tcl (if $LIVEDRIVE stands for the CD, which is already mounted at this point in my code, and $PUPBUILD is the /puppylivecdbuild directory) that would be:

Code: Select all

set cd_files [glob -directory $LIVEDRIVE *]
foreach f $cd_files {
	if {[regexp "boot.cat|pup_\[0-9\]+?.+?\.sfs" $f] == 0} {
		file copy -force $f $PUPBUILD
	}
}
Thanks--that's the last finishing touch! I'm now running Puppy 4.1.1 from a live-CD I burned with Remaster Express, and the total size is less than official Puppy 4.1.1 because of uninstallation of built-in packages. I'll package up the 4.1.1 version and release it at the first opportunity. Then I'll see if I can get the 4.2 Beta packages.txt and 0pkgs_db.pet, and produce a 4.2 version--which, if all goes well, may have the honor of being the first new package for Puppy 4.2 that missed the beta deadline! :roll:
It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
pa_mcclamrock
Posts: 695
Joined: Fri 03 Jun 2005, 23:13
Location: Fort Wayne, Indiana, USA

#16 Post by pa_mcclamrock »

It's stupid to use inferior software for ideological reasons.
--Linus Torvalds

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#17 Post by technosaurus »

I remastered it for myself to do some extra stuff and so that I could more easily modify it...I found it difficult to understand some of the stuff in the other script

Mine is run by dragging an empty folder to it from a separate partition... so I took out all of the finding best partition stuff and simplified it a lot
this was mainly just for me to learn Xdialog and bash so a lot of the simplifications were more out of my understanding so that I could modify it.... has a few extra options that I couldn't just add to remasterpup and has worked for me so far
Attachments
remasterwizard.tar.gz
here is my version if you want to use parts
(2.84 KiB) Downloaded 384 times
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
Max Headroom
Posts: 421
Joined: Wed 28 Jun 2006, 07:17
Location: GodZone Kiwi
Contact:

technosaurus

#18 Post by Max Headroom »

technosaurus, Please Update Us on Progress of Y'r Planned MultiBoot SuperPup, ETA.
Can't Wait! :)

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#19 Post by technosaurus »

@max
funny you should post that here - the method I was using to allow more puplets on a disk causes some issues (using the addon sfs as the zdrv sfs)... My script here allows you to combine the pup and zdrv into one sfs (as well as any loaded sfs and installed pets) and then to remaster it into an iso... the default install/remaster wizard doesn't play well with this unconventional setup so I at least wanted a way for people to install their choice puplet without having to redownload it
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
Max Headroom
Posts: 421
Joined: Wed 28 Jun 2006, 07:17
Location: GodZone Kiwi
Contact:

Thanx 4 the reply technosaurus

#20 Post by Max Headroom »

Thanx 4 the reply technosaurus, altho I'm not sure I'm understanding it :? Does that mean U'r Abandoning Y'r Ambitous SuperPup?

Post Reply