Auto-build a Puppy iso; single script with optional gui

[Keef]

wiak

Code: Select all

2createpackages CHANGED_ONLY

I think you maybe right about this. Some time ago there was a debate about this subject, and I found the same option myself. It is not documented (or wasn't then), and got no reaction when I posted about it, despite the more convoluted methods people were using to avoid rebuilding packages.
Anyway, it did work for me. Not tried your script yet, but will give it a try out soon. Internet very unreliable at the moment so downloading is a bit of a pain.

[EDIT] Yes that was the option - found my post buried in the Slacko Woofers thread,

[wiak]

wiak

Code: Select all

2createpackages CHANGED_ONLY

...

[EDIT] Yes that was the option - found my post buried in the Slacko Woofers thread,

Thanks Keef, I wish I had noticed your post on Slacko woof threads (surprised no one commented on your post, not even 01micko, or picked the tip up - would have been better that it was fully confirmed/documented at that stage since would have saved others this later fluffing-about time...), but I saw the mention of CHANGED_OPTION in 2createpackages and wondered about it but wasn't entirely sure of its purpose till I read through the code - and wanted to do that anyway (or get confirmation from jlst or someone who likely knows) before I could risk modifying makepup on the basis. Anyway, as I say, the run_findpkg routine, EDIT: which is in the _00func script uses support/findpkgs script to check md5sums and so on, and testing suggests it works. Thanks for your confirmation and double-check.

I'm wondering now what other options may be buried in the code or if they are documented outside the code somewhere? Its possible some code options have come from BarryK, and not been noticed as yet (seems unlikely), though the name of that option did suggest the possible purpose.

EDIT: have now found you old post. I see you suggest it has been mentioned somewhere before, though I haven't come across earlier references as yet:

http://www.murga-linux.com/puppy/viewto ... 014#870014

It certainly was a bit obscure because tricky reading the code to find its purpose since relies on _00func interaction, involving DISTRO_PKGS_SPECS that is quite involved. Very useful piece of code though.

EDIT2: I've now found another relevant link via google puppy searching (shows what a long history option CHANGED_ONLY actually has...):

http://murga-linux.com/puppy/viewtopic. ... 841#368841

EDIT3: I've now finally gotten round to looking at script woof-gui and this is mentioned:

Code: Select all

      <hbox>
        <text use-markup="true"><label>"<b>This tab is for creating the cut-down packages needed to build a Puppy live-CD</b>. This tab is a frontend for script '2createpackages'"</label></text>
        <text><label>You must have previously downloaded all the packages, see 'Download pkgs' tab. Now, they need to be processed into a cut-down and Puppy-compatible format. The resulting packages are created in directory 'packages-${DISTRO_FILE_PREFIX}'</label></text>
      </hbox>
      <hbox>
        <text><label>'ALL_PACKAGES' means all chosen packages (see 'Choose pkgs' tab) are processed, directory 'packages-${DISTRO_FILE_PREFIX}' is wiped and all packages re-created. 'CHANGED_ONLY' will automatically process only packages that have changed or added since last time 2createpackages was run. Or, if only one package has been added or modified in the 'Choose pkgs' tab, then it can be explicitly selected here to process just that one:</label></text>
        <combobox><variable>CHOICE_CREATEPACKAGES</variable>${ITEMS_CG}</combobox>
      </hbox>

though I haven't checked as yet where CHANGES_ONLY option is used by the underlying code there.

EDIT4: Ah, here is where woof-gui optionally uses that CHANGES_ONLY option:

Code: Select all

  <button><label>CREATE PACKAGES</label><action>rxvt -title "Creating packages" -bg yellow -e ./2createpackages \$CHOICE_CREATEPACKAGES</action><action>xmessage -bg green -center "Finished. Click 'REPORT' button for a summary."</action></button>

wiak

[rufwoof]

Changed makepup line 336 to include CHANGED-ONLY parameter and re-ran my testbed for the third time (far from sure that's the correct choice, but gave it a try anyway).

Code: Select all

else # no user input required
 merge2out_mod # or could implement all this using tcl expect package (and autoexpect) instead, but big install...
 ./merge2out_wiak
 sleep 1 # maybe remove, or adjust duration
 cd ../woof-out_*
 cp -af DISTRO_PKGS_SPECS-* ../ || { printf "\n\nError: DISTRO_PKGS_SPECS-distro-version is missing.\n\nCheck settings and try again. Exiting...\n"; exit 1; }
 # DISTRO_PKGS_SPECS-${DISTRO_BINARY_COMPAT}-${DISTRO_COMPAT_VERSION} is ready for reading at this stage
 # So use -p/--pause flag here to pause script here to allow manual editing of makepup_extra.conf extra packages list.
 pause4adding_packages
 0setup_mod
 1download_mod
 2createpackages_mod CHANGED_ONLY
 3builddistro-Z_mod

and took 47 mins to run through which includes building the devx.sfs. Started building the mksquashfs after 30 minutes, which took 8 mins to run through (a lighter choice of compression could create a sfs in less time, but would be larger).

So no noticeable time difference from my second run

[wiak]

Changed makepup line 336 to include CHANGED-ONLY parameter and re-ran my testbed for the third time (far from sure that's the correct choice, but gave it a try anyway).

Code: Select all

else # no user input required
 merge2out_mod # or could implement all this using tcl expect package (and autoexpect) instead, but big install...
 ./merge2out_wiak
 sleep 1 # maybe remove, or adjust duration
 cd ../woof-out_*
 cp -af DISTRO_PKGS_SPECS-* ../ || { printf "\n\nError: DISTRO_PKGS_SPECS-distro-version is missing.\n\nCheck settings and try again. Exiting...\n"; exit 1; }
 # DISTRO_PKGS_SPECS-${DISTRO_BINARY_COMPAT}-${DISTRO_COMPAT_VERSION} is ready for reading at this stage
 # So use -p/--pause flag here to pause script here to allow manual editing of makepup_extra.conf extra packages list.
 pause4adding_packages
 0setup_mod
 1download_mod
 2createpackages_mod CHANGED_ONLY
 3builddistro-Z_mod

)he devx.sfs. Started building the mksquashfs after 30 minutes, which took 8 mins to run through (a lighter choice of compression could create a sfs in less time, but would be larger).

So no noticeable time difference from my second run

No, it won't work that way EDIT: 2createpackages_mod is a function that simply modifies woof-CE original 2createpackages into 2createpackages_wiak. The one you should put CHANGED_ONLY on as option is thus "2createpackages_wiak", which gets called up in makepup. I'll post modified version of makepup that includes --update switch shortly, which does work

If you want to just test more quickly, try changing the makepup line 345 or thereabouts:

Code: Select all

./2createpackages_wiak CHANGED_ONLY

I think that should probably work (maybe...), but as I say I'll probably want to keep non-CHANGED_ONLY behaviour too so 'maybe' planning a new commandline --update/-u switch (or perhaps, more likely maybe, a switch to force change ALL_PACKAGES).

wiak

[rufwoof]

Thanks wiak.

I've just made that edit and also set devx creation to false and fired off another run. Unlike prior runs however this one was started with the woof-out_x86_64_x86_64_ubuntu_xenial64 folder deleted.

Personally I'm happy with things as-is (single click <1 hour to build a version updated for all of the latest changes/security patches) Takes Puppy into a new dimension.

[rufwoof]

Around 10 mins in and its flying through. Done the 1st and 2nd passes and now just running through the 'Processing ...' where it looks by eye that all of the .DEB files are being seen as unchanged (not being processed), but where all .pet's seem to be being processed (as though they've changed). Again I'm in the dark here so might just be my misinterpretation.

[wiak]

Is there a way provided by 1download to indicate which packages have been updated such that I can automate 2createpackages to selectively operate on just these packages in terms of update?

LOL - I've got a tweak to 1download that I will submit to woof-ce tomorrow for precisely this....

Hi Peebee,

Yes, fair enough to add code to woof-CE 1download script in order to also get a download_list though. I did also notice md5sums don't seem to actually be stored anywhere for non-pets, so a download_list could also be speedier than any existing methods (assuming CHANGED_ONLY method actually works as of now.

wiak

[wiak]

Around 10 mins in and its flying through. Done the 1st and 2nd passes and now just running through the 'Processing ...' where it looks by eye that all of the .DEB files are being seen as unchanged (not being processed), but where all .pet's seem to be being processed (as though they've changed). Again I'm in the dark here so might just be my misinterpretation.

Thanks rufwoof,

Looking forward to your further update on the final result. I'm also a bit in the dark and hence waiting to release new makepup based on any of this; the key all lies in the woof-CE support/findpkgs script, which I haven't worked out its full operation as yet so don't know if CHANGED_ONLY will do as we wish as things stand (may or may not...). support/findpkgs script is itself around twice the size of makepup...!

Peebee's 1download mod to woof-CE itself might prove to be a better method in the end.

support/findpkgs actually seems to be checking out modification dates (I'm not sure) rather than just md5sums (actually, on re-reading the script, it seems to use md5sums instead of actual modification dates) but I haven't worked out the details and the mechanism may not be very efficient as done:

Code: Select all

## DS_DATE and DPS_MODIFY_DATE are md5 checksums
## when these are different it means an update has to happen

wiak

[rufwoof]

Not sure what happened with that last run of mine. I had set build devx off ... but it still built one. Noticed that I have two CPU temperatures in the panel now so I think running multiple times in a single session (sandboxes ...etc.), might all be getting tangled up in some way.

I'll clean up that testbed and start afresh from a clean reboot and just makepup alone ... but another day (not tonight).

[davids45]

G'day wiak,

Thanks for your previous responses to my problem getting makepup working fully.

I don't know what was going wrong with my initial make-pups but by moving/copying the make-pup directory from my Frugal Slacko-6.9.9.9 root directory to my data partition and running the script in that from a Full Xenial Pup, I achieved an iso of the default slacko .

Next evening, hubris to the fore, I made a Xenial Pup, again without problem . Both took order-of-magnitude 1 hour.

Today, I ran both isos (via manual-install copying of their boot directory files) as new Frugals and both were fine.

Just the common problems in woof-CEs due to missing libs (e.g. libpng.so.12 in slacko) and incompatible/buggy (?) ghostscript files so printing doesn't work - I'll try the "use-the-old gs" files fix later. Scanning is also a variable problem - some packages work, another previews OK but trying an actual scan, it tells me I "don't have enough disk space" which is a bug in that particular package.

When I find the time, I'll see if I can fix these woof-CE issues I have by tinkering with the makepup files.

Thanks again for makepup.

David S.

[wiak]

Hello David,

Yes, I was going to ask if you were trying to build on your Puppy save file rather than somewhere on /mnt/home. Yes, that would explain the issues you were having - your builds were running out of resource space and hanging....

Good to hear all good now.

wiak

[wiak]

Hi all,

I have just completed a test of using CHANGES_ONLY on the second run of makepup and all went well and the second build (since it had nothing to download or rebuild (aside from some minor config matters and the squashfs itself) successfully completed the update in the order of 15min...

My first run was in an empty folder using default version 0.0.2:

Code: Select all

./makepup

That successfully build the default Slacko pup as normal.

I then modified makepup by changing line 348 from:

Code: Select all

./2createpackages_wiak

to

Code: Select all

./2createpackages_wiak CHANGED_ONLY

(leaving all the previous build directories untouched)

and that just zipped through most everything and the 'less than 15 minutes to create' update/rebuild seems to be fine.

More testing required, but this CHANGED_ONLY method seems to be fine (though can't be sure since no packages have needed updated as yet, so can't test if it will work that I can think of). Oh, I know, I'll remove one of the packages from my local-repositories and run that modified makepup again to see if it picks that up and re-installs the missing package...

I'll report back in half an hour or whatever it all takes.

wiak

wiak

[wiak]

Okay, I removed the previously downloaded files concerned with busybox and also those concerning autologin from local-repositories/x86/packages-pet. I also removed bash and coreutils from local-repositories/x86/packages-tgz_txz-14.2. And then ran the makepup containing CHANGED_ONLY line again.

It successfully noticed the missing files and quickly replaced them and finished the overall rebuild once again in under 15 minutes.

I'll do one more test for now. This time I'll damage a couple of files in these local-repository folders (change a few bytes with a hex editor after first manually checking their md5sums). Then I'll see if running CHANGED_ONLY makepup successfully notices that and fixes everything in that hoped for less than 15 min again. Reporting back soon.

EDIT: Actually I used a combination of /dev/random and dd to change one byte in both busybox....pet and in bash...txz. I could of course simply have used a different file as a replacement for busybox (renaming to busybox etc, which would also guarantee a different md5sum from the original of course... same with bash and any other program I wanted to 'corrupt').

EDIT2: So the answer to the above is NO... - re-running makepup with CHANGES_ONLY mod did not fix either the md5sum corrupted busybox dotpet nor the bash txz. So my understanding of how it might work isn't correct. However, might still be fine if makepup with CHANGES_ONLY addition is able to update when the package has a different version number (?). I'll try now modifying the busybox dotpet and bash txz package names with different version numbers than their originals and see if they get auto-replaced. Otherwise I can't think of a way of testing if auto-updating with CHANGES_ONLY usefully works... (well, at least we know now it DOES fix missing local-repository packages, but not md5sum corrupted ones... maybe it does somehow check file modification dates?) until such time as an actual package is known to have been updated up-stream. Again, I'll report back soon.

EDIT3: I changed the version numbers of the local-repository stored files for autologin (pet), busybox (pet), bash (txz) and coreutils (txz) and on running makepup with CHANGED_ONLY mod the correct current versions were downloaded (the changed ones weren't deleted from inside local-repositories). and then the build proceeded successfully (total build time again less than 15 minutes on my Intel Core II duo, 2 GHz CPU, 2 GB quite old laptop and very slow broadband of less than 500 kB/sec). According to file woof-installed-packages, everything now seems to be fine again. If that is so then CHANGED_ONLY method seems to work, though I have no idea how it actually determines what needs updated... More testing required for better certainty, but looking good...

wiak

[wiak]

OK, my opinion is that CHANGES_ONLY update method works fine, and fast.

I'll implement that in next makepup in some manner or other. Probably upload version 0.0.3 with that change sometime next day or two.

If that method ends up not working so well as it seems to (or I hope) I can always use alternative method (based on list of new 1download packages) in later makepup version.

wiak

[wiak]

Tip:

If you have tons of RAM on your system, try putting a copy of the makepup script in /tmp and build your Puppy in there. Should be much faster than building to a harddrive. Having said that, this tip will not improve bottleneck of download speed (the issue I have with my rural broadband connection).

Got the idea behind that tip from this nice woof building tutorial by Iguleder, which I came across only today:

http://www.murga-linux.com/puppy/viewto ... ppage=2000

However, I don't have much RAM (sad) so I can't test the tip.

Iguleder's tutorial is very old by the way (read as 'ancient'), so read it more generally from that out-of-date point of view.

More up-to-date info in thread below, but not much technical detail:

http://puppylinux.com/woof-ce.html

And remember: makepup requires no knowledge of git, github and DOES NOT require git to be installed on your puppy system (and DOES NOT need a devx loaded unless you need binaries stripped) - just needs your standard, relatively recent Pup for running makepup script on... I'm hoping it provides a relatively simple introduction to building a woof-CE Puppy...

I don't know if there is much documentation of the internal technical aspects of woof-CE operation actually. Wish I did rather than just reading the scripts in there!

Fortunately, you don't need to know any of this technical stuff to build a woof-CE puppy using makepup script. Just use the script and that happens automatically for you...

I'm planning a short rest now before proceeding with version 0.0.3 of makepup.

wiak

[rufwoof]

Hi wiak. When I very cursory glanced over some of the code my mental image was that it pulls down list(s) of what's in the corresponding puppy, validates the md5 checksum of those lists and then wades through those lists ... if the corresponding file already exists in the local repository then doesn't download, otherwise it is downloaded ... without md5 checking each. My fluency in reading 'script' however is a lot lower than yours, so I'm likely wrong.

If a package/lib is changed then generally it tends to have a new filename (version) ... often a dash version for instance in Xenial there's a perl_5.22.1-9_amd64.deb suggesting to me that version 5.22.1 of that has been updated/changed 9 times.

What I don't know is how the main filelist files are maintained. I'm guessing that if for instance Ubuntu/Debian release a perl_5.22.1-10 then there could be a automated script that perhaps runs nightly to change the main list within woof-CE so that's reflected through? I suspect perhaps not however and that it takes a developer to select what is in the Puppy so as to keep it more stable (unchanging), so ... for instance, a later perl_5.22.1-10 rolled out by Ubuntu might not be updated in woof unless specifically selected by a developer as the version in woof for puppy xyz. I'm hoping for the former, but suspecting the latter.

[wiak]

Hi wiak. When I very cursory glanced over some of the code my mental image was that it pulls down list(s) of what's in the corresponding puppy, validates the md5 checksum of those lists and then wades through those lists ... if the corresponding file already exists in the local repository then doesn't download, otherwise it is downloaded ... without md5 checking each. My fluency in reading 'script' however is a lot lower than yours, so I'm likely wrong.

If a package/lib is changed then generally it tends to have a new filename (version) ... often a dash version for instance in Xenial there's a perl_5.22.1-9_amd64.deb suggesting to me that version 5.22.1 of that has been updated/changed 9 times.

What I don't know is how the main filelist files are maintained. I'm guessing that if for instance Ubuntu/Debian release a perl_5.22.1-10 then there could be a automated script that perhaps runs nightly to change the main list within woof-CE so that's reflected through? I suspect perhaps not however and that it takes a developer to select what is in the Puppy so as to keep it more stable (unchanging), so ... for instance, a later perl_5.22.1-10 rolled out by Ubuntu might not be updated in woof unless specifically selected by a developer as the version in woof for puppy xyz. I'm hoping for the former, but suspecting the latter.

Actually, rufwoof, my 'understanding' of the md5 checking of the package lists is much the same as you write above (the md5 checks are on the pkg lists). And my 'worries' about these package lists (and lack of knowledge about them) is currently identical to yours. This is part of the reason I'm taking a rest to get some energy back cos need to read more of these extremely long scripts to see if there is any clue about that. Would be a real pity if these package lists do need to be manually updated every so often on woof-CE itself, but I also wonder how it can be getting done otherwise... (my hope is that woof-CE scripts actually download package lists somehow from up-stream Debian/Ubuntu etc, though I don't know if that info is available from there - maybe you know?) Anyway, I guess 0setup, 1download, _00func, and support/findpkgs scripts (along with the stuff they source) may contain the answers to these questions... Time will tell. Still fun to play around with though (and exercise for the brain).

EDIT: most useful and time-saving would be if jlst or 01micko will tell us or if we ask them of course. Will do so if I can't work it out myself soon, or you don't either.

EDIT2: This is one of the advantages in debootstrap in Debian of course. Potentially big team of developers and since it is from Debian everything being kept up to date. Makes scripting Debian-based systems relatively simple really.

EDIT3: I suppose if you have an actual Debian Stretch system you can apt-get update packages and compare with what Puppy woof-CE provides and see if any and all packages are up-to-date or not... Like you, I doubt it. If not, then that is main thing that Puppy developers/woof-CE-stewards need to work on. Or alternatively, forget Debian/Slackware etc compatibility and go back to trying to build smaller Puppy using its own repository via woof-CE (like tiny core and Slitaz do) - yes, disadvantage then of lots of build work and small repository, but maybe advantage of smaller size possibility? I know hardware resources tend to be large now anyway, so small distribution size not often so important, but sometimes nice anyway (and certainly on older laptops/machines). Maybe should see if possible to woof from tiny core or Slitaz repos actually... but again package lists need to be kept somehow up to date.

EDIT4: Could be this: support/debdb2pupdb but this is a compiled binary, so need to find/read the original source code. Actually, I'm becoming hopeful that DISTRO-PKGS-SPECS-<distro-version> information is indeed downloaded form upstream debian (or whatever main distro is) and for example, for slackware, related to this file: DISTRO_COMPAT_REPOS-slackware-14.

For example, in there (along with other possible repos) we have:

Code: Select all

PKG_DOCS_DISTRO_COMPAT="
z|http://ftp.nluug.nl/pub/os/Linux/distr/salix/${DBIN_ARCH}/slackware-${DISTRO_COMPAT_VERSION}/PACKAGES.TXT

The above resolves to the package database list file:
http://ftp.nluug.nl/pub/os/Linux/distr/ ... CKAGES.TXT

If you go to that url you should see the package details and I think that is likely downloaded and processed by 0setup and code called from that.

If you build a Debian Stretch Puppy instead and look in DISTRO_COMPAT_REPOS... I expect you'll find the url to the similar package database file for Debian Stretch which 0setup will automatically download and process when building Pup based on that distribution etc...

In other words, I'm becoming pretty sure 0setup script is automatically updating the distro package lists, in which case auto-updates should work fine. [/b]Maybe I'm just an optimist (!) and nothing to do with makepup development really, so I'll just move on with that for now.

The following and more in 0setup:

Code: Select all

#download docs on compatible-distro pkgs...
function download_compat_pkg_dbs() { etc...
...
	  if pkg_db_confirm_download ; then
	   DLFILE="`basename $PKGLISTURI`"
	   [ -f $DLFILE ] && mv -f $DLFILE /tmp/${DLFILE}-backup1 #v431 otherwise wget creates a new file ${DLFILE}.1
	   [ "$DBmethod" = "true" ] && rxvt -name pet -bg orange -geometry 80x10 -e wget $PKGLISTURI || wget $PKGLISTURI
etc...

wiak

[wiak]

Okay, I'm fairly satisfied there is no problem in making CHANGED_ONLY the default for 2createpackages_wiak line (whether using makepup from an empty directory or after a previous build). Unless anyone's further tests suggest otherwise about that, I'll code a switch -u (or --update) to change that behaviour of 2createpackages to ALL_PACKAGES, which will be a method to force the re-download and rebuild of everything. Otherwise only changed packages should be downloaded by default and rebuilt.

Will be in version 0.0.3 which I'll upload tomorrow sometime probably.

I've also almost convinced myself that the upstream package database pkg lists are being updated on a re-build, which is fantastic if true!

Kudos to woof-CE developers. Puppy forum members need to focus more on helping out with woof-CE in my humble opinion...

wiak

[rufwoof]

If woof is a set of generic package names and pointers to repositories (such as Debian), then pulling down a package will automatically pull down the latest version ... as maintained by Debian (whoever) in their own repository.

When however there is a local repository with a version already in that, then CHANGED_ONLY might see a version already locally available and not download from Debian (wherever) ... whilst the versions could be different.

I'd guess that its perhaps best to clear out the local repository for each run if you were looking to use makepup as a latest versions tracker. Or perhaps add additional tests/code to test the local repository versions with that in the Debian repository. Debsums is a debian utility that validates local versions of packages compare to that in the repository and it runs through really quick so if not already catered for within woof for something like that it would seem to be a relatively low overhead (time) and would be useful for those on low bandwidth (less of a issue for those on higher bandwidth).

[wiak]

If woof is a set of generic package names and pointers to repositories (such as Debian), then pulling down a package will automatically pull down the latest version ... as maintained by Debian (whoever) in their own repository.

When however there is a local repository with a version already in that, then CHANGED_ONLY might see a version already locally available and not download from Debian (wherever) ... whilst the versions could be different.

I'd guess that its perhaps best to clear out the local repository for each run if you were looking to use makepup as a latest versions tracker.

Perhaps... in which case would use --update meaning ALL_PACKAGES. However, I have a feeling (not confirmed) that the md5 package list check routines in support/findpkgs forces download of new package lists if md5sums in newer versions upstream do not match in which case CHANGED_ONLY will work perfectly. EDIT: Actually findpkgs routine call comes after 0setup, so what I suggest there isn't correct. Have to look more closely at 0setup arrangement itself to see if anything there - or best just to try it!

Anyway, makepup will contain -u option to allow test of both possibilities.

Please refer to a few EDITS (particularly EDIT4) I made to my post (two posts above) which illustrates where I believe the package lists are retrieved from upstream by 0setup script.

wiak