Auto-build a Puppy iso; single script with optional gui

A home for all kinds of Puppy related projects
Message
Author
User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#46 Post by rufwoof »

Thanks wiak

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#47 Post by peebee »

wiak wrote:
rufwoof wrote:
Do you know whether if you leave local-repositories will the next run pull down any changed versions of those i.e. later versions of libs/programs if any are available?
Yes - latest versions will be downloaded by ./1download as long as repositories have been updated by ./0setup....

./2createpackages can then be run selectively on just the updated packages.

This makes building an updated system very fast (c. 15mins for LxPupSc).
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#48 Post by rufwoof »

Thanks peebee.

Just built a 64 bit Xenial from woofce-rationalise (I hard coded makepup values to ...

Code: Select all

# Default config:
nTARGETARCH="3"              # i.e. target is x86 (1=arm, 2=x86, 3=x86_64)
nCOMPATDISTRO="5"            # i.e. distro is slackware (1=debian, 2=devuan, 3=slackware, 4=trisquel, 5=ubuntu)
nCOMPATVERSION="2"           # 2 is xenial (1 trusty)
WOOFBRANCH="woof-CE-rationalise" # woofbranch, e.g. woof-CE-testing
#WOOFBRANCH="woof-CE-testing"
nHUGEKERNEL="28"             # i.e. hugekernel to use huge-4.9.15-xenialpup64.tar.bz2
interactive="false"          # "true" means: Require user input during the build
gui="false"                  # "true" means: Use gui frontend (wiak remove comment when done: gui not yet implemented)
keep="false"                 # "true" means: Keep previous woof download files and directories
pause="false"                 # "true" means: pause makepup script just after merge2out routine to allow package selection changes
DEVX="true"                 # "true" means: Create DEVX sfs
Built and booted fine.
Image
Sound, internet, change resolution, firewall, savefolder ...etc. all working as expected. Only problems I've found so far is that the drives top right unmount hot area doesn't work (just opens the drive in rox) and using pupclockset to change the trays time format to a custom %a %d %b %R ... and then saving, had the top of screen panel/tray lose its settings to leave just setup and jwmdesk icons only in that panel ... and no minimised windows shown either (I'm having to use Ctrl-Alt-Tab to switch between windows as I post this).

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#49 Post by wiak »

peebee wrote: Yes - latest versions will be downloaded by ./1download as long as repositories have been updated by ./0setup....

./2createpackages can then be run selectively on just the updated packages.

This makes building an updated system very fast (c. 15mins for LxPupSc).
He Peebee, rufwoof and anyone reading,

Thanks for that info, Peebee. It's especially useful to have such input from an experience Puppy builder like yourself. I can easily add a --update/-u switch to makepup to modify its behaviour for update packages use. Is there a way provided by 1download to indicate which packages have been updated such that I can automate 2createpackages to selectively operate on just these packages in terms of update? A step by step example of how you go about such an update, including how you use 2createpackages to selectively operate would help me automate these mechanisms via makepup function.

Otherwise I can use modification times to local-repository files as a possible mechanism but still need to know how to use 2createpackages for selective operation.

Such modifications to makepup are likely to be made later anyway since other improvements still in progress.

wiak
Last edited by wiak on Wed 30 Aug 2017, 21:36, edited 2 times in total.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#50 Post by rufwoof »

I've just run makepup for a second time, leaving everything from the first run (that took about a hour) as-is, and the second run took 48 minutes (and seems to have built the iso and devx ok). Seems about right as I'm on a 100Mbs link so I guess around 12 minutes of download time for the first run not being required in that second run.

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#51 Post by wiak »

rufwoof wrote: I'm on a 100Mbs link
You are fortunate, rufwoof. I live in a rural area that only has very slow broadband (around 0.5Mb download speed...) so each makepup build test takes a while - though still less than a couple of hours even when using an empty local-repository...

wiak

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#52 Post by peebee »

wiak wrote:Is there a way provided by 1download to indicate which packages have been updated such that I can automate 2createpackages to selectively operate on just these packages in terms of update?
LOL - I've got a tweak to 1download that I will submit to woof-ce tomorrow for precisely this....
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

User avatar
peebee
Posts: 4370
Joined: Sun 21 Sep 2008, 12:31
Location: Worcestershire, UK
Contact:

#53 Post by peebee »

rufwoof wrote:I've just run makepup for a second time, leaving everything from the first run (that took about a hour) as-is, and the second run took 48 minutes (and seems to have built the iso and devx ok). Seems about right as I'm on a 100Mbs link so I guess around 12 minutes of download time for the first run not being required in that second run.
Most of that is probably a repetition of a complete 2createpackages that wasn't really needed....
ImageLxPup = Puppy + LXDE
Main version used daily: LxPupSc; Assembler of UPups, ScPup & ScPup64, LxPup, LxPupSc & LxPupSc64

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#54 Post by rufwoof »

peebee wrote:Most of that is probably a repetition of a complete 2createpackages that wasn't really needed....
Yes peebee. I just sat and watched it do all of that. Just a test really to see if I got the same OK outcome result for Xenial as wiak got for Slacko when repeat running without having cleaned things out.

Should also say I'm running a AMD Phenom x4 2GB that's around 10 years old so my timings are relative to that. Using a ext3 format partition (otherwise empty). [I like ext3 as that's a more flexible choice i.e. BSD can rw to that, but as though ext2 (no journaling), whilst it can be mounted as though a ext4 (for even better recovery/data safety, however ext3's stability/recovery is pretty good anyway IME)].

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#55 Post by rufwoof »

That second Xenial 64 build works fine. Looks like a lot of work going into the next Puppy and there are issues with using the menu based configuration tools, however reverting to a manual edit approach of jwm files and otherwise its working really well.

Image

Image

... and with makepup tracking updates is just a click away :)

User avatar
recobayu
Posts: 387
Joined: Wed 15 Sep 2010, 22:48
Location: indonesia

#56 Post by recobayu »

This is what i got. I make Dpup Stretch using this makepup.
I must download firefox from ppm first because the defaultbrowser can not walk. Very Nice, Wiak and Woof-CE Community..
Thank You.

Image

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#57 Post by wiak »

Thanks for your report recobayu!

Glad to hear makepup script has worked for you making a Dpup Stretch.

Could you kindly post the makepup commandline you used since that info proves to be a useful resource, so is always useful in reports if only for confirmation of makepup commandline options used.

It's also good to know specification of the build machine you used, Puppy host distribution (name, version and whether 32bit or 64bit) and whether 32bit or 64bit target distribution. An indication of the time it took for the build to complete would also be great. A lot to ask, I know.... :-)

For your commandline, I'm presuming:

Code: Select all

./makepup -d 1 -r 1
with no other commandline options added?

Thanks in advance if you can provide any of that info!

wiak

User avatar
recobayu
Posts: 387
Joined: Wed 15 Sep 2010, 22:48
Location: indonesia

#58 Post by recobayu »

Code: Select all

./makepup -d 1 -r 1
Yes. But it is very loongg time to wait it.
1270 items in folder packages-deb-stretch, and
153 items in packages-pet.

My suggestion is, There is better if after make an iso (or remaster from that iso) and devx, someone upload that iso so we can download it directly.

I copied my jwmrc-tray from xenial to change the tray full horizontall.

here is the iso (238M):
https://drive.google.com/file/d/0B139JQ ... sp=sharing
Attachments
hardinfo_report.html.gz
(84.07 KiB) Downloaded 158 times

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#59 Post by wiak »

recobayu wrote:

Code: Select all

./makepup -d 1 -r 1
Yes. But it is very loongg time to wait it.
1270 items in folder packages-deb-stretch, and
153 items in packages-pet.
Hi recobayu,

Yes, takes a long time, though not quite so long if you need to rebuild/update later and that is being worked on.

However, makepup is basically a frontend for woof-CE to make that a one-click easy operation and the purpose of woof-CE is to help easily produce an initial Puppy iso by anyone who feels like it - people can thereafter refine what they produce for their own use or to publish as an iso later. It all depends therefore what you want to use makepup/woof-CE for - some will want to simply use it to build an iso for their own use so the time to build is not so important then.

Thanks again for you report - it is good to see woof-CE being used and successfully. And thanks for sharing your iso for whoever wants to try it.

wiak

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#60 Post by wiak »

peebee wrote:
wiak wrote:Is there a way provided by 1download to indicate which packages have been updated such that I can automate 2createpackages to selectively operate on just these packages in terms of update?
LOL - I've got a tweak to 1download that I will submit to woof-ce tomorrow for precisely this....
Hi peebee,

re: looking at your woof-CE Pull request:

I see. So your plan is to have the name only of any new package downloaded stored under status/download_list and then use that list with 2createpackages to minimise the amount of work it needs to do? Maybe...

However, I've not quite worked this out yet, but the code in 2createpackages seems to have an option to only create changed packages (it appears to use run_findpkgs to sort out what has been changed based on md5sums...). 1download script uses the exact same run_findpkgs routine as far as I can see.

The option I'm taking about is:

Code: Select all

2createpackages CHANGED_ONLY
If that is correct, then your proposed 1download change would seem unnecessary. But I haven't had time to check if using that CHANGED_ONLY option with 2createpackages does what I think it does... I'll try it tomorrow. If it does then it is very easy for me to add a --update/-u switch to makepup to include that facility.

wiak

User avatar
Keef
Posts: 987
Joined: Thu 20 Dec 2007, 22:12
Location: Staffordshire

#61 Post by Keef »

wiak

Code: Select all

2createpackages CHANGED_ONLY
I think you maybe right about this. Some time ago there was a debate about this subject, and I found the same option myself. It is not documented (or wasn't then), and got no reaction when I posted about it, despite the more convoluted methods people were using to avoid rebuilding packages.
Anyway, it did work for me. Not tried your script yet, but will give it a try out soon. Internet very unreliable at the moment so downloading is a bit of a pain.

[EDIT] Yes that was the option - found my post buried in the Slacko Woofers thread,

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#62 Post by wiak »

Keef wrote:wiak

Code: Select all

2createpackages CHANGED_ONLY
...

[EDIT] Yes that was the option - found my post buried in the Slacko Woofers thread,
Thanks Keef, I wish I had noticed your post on Slacko woof threads (surprised no one commented on your post, not even 01micko, or picked the tip up - would have been better that it was fully confirmed/documented at that stage since would have saved others this later fluffing-about time...), but I saw the mention of CHANGED_OPTION in 2createpackages and wondered about it but wasn't entirely sure of its purpose till I read through the code - and wanted to do that anyway (or get confirmation from jlst or someone who likely knows) before I could risk modifying makepup on the basis. Anyway, as I say, the run_findpkg routine, EDIT: which is in the _00func script uses support/findpkgs script to check md5sums and so on, and testing suggests it works. Thanks for your confirmation and double-check.

I'm wondering now what other options may be buried in the code or if they are documented outside the code somewhere? Its possible some code options have come from BarryK, and not been noticed as yet (seems unlikely), though the name of that option did suggest the possible purpose.

EDIT: have now found you old post. I see you suggest it has been mentioned somewhere before, though I haven't come across earlier references as yet:

http://www.murga-linux.com/puppy/viewto ... 014#870014

It certainly was a bit obscure because tricky reading the code to find its purpose since relies on _00func interaction, involving DISTRO_PKGS_SPECS that is quite involved. Very useful piece of code though.

EDIT2: I've now found another relevant link via google puppy searching (shows what a long history option CHANGED_ONLY actually has...):

http://murga-linux.com/puppy/viewtopic. ... 841#368841

EDIT3: I've now finally gotten round to looking at script woof-gui and this is mentioned:

Code: Select all

      <hbox>
        <text use-markup="true"><label>"<b>This tab is for creating the cut-down packages needed to build a Puppy live-CD</b>. This tab is a frontend for script '2createpackages'"</label></text>
        <text><label>You must have previously downloaded all the packages, see 'Download pkgs' tab. Now, they need to be processed into a cut-down and Puppy-compatible format. The resulting packages are created in directory 'packages-${DISTRO_FILE_PREFIX}'</label></text>
      </hbox>
      <hbox>
        <text><label>'ALL_PACKAGES' means all chosen packages (see 'Choose pkgs' tab) are processed, directory 'packages-${DISTRO_FILE_PREFIX}' is wiped and all packages re-created. 'CHANGED_ONLY' will automatically process only packages that have changed or added since last time 2createpackages was run. Or, if only one package has been added or modified in the 'Choose pkgs' tab, then it can be explicitly selected here to process just that one:</label></text>
        <combobox><variable>CHOICE_CREATEPACKAGES</variable>${ITEMS_CG}</combobox>
      </hbox>
though I haven't checked as yet where CHANGES_ONLY option is used by the underlying code there.

EDIT4: Ah, here is where woof-gui optionally uses that CHANGES_ONLY option:

Code: Select all

  <button><label>CREATE PACKAGES</label><action>rxvt -title "Creating packages" -bg yellow -e ./2createpackages \$CHOICE_CREATEPACKAGES</action><action>xmessage -bg green -center "Finished. Click 'REPORT' button for a summary."</action></button>
wiak
Last edited by wiak on Thu 31 Aug 2017, 22:24, edited 4 times in total.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#63 Post by rufwoof »

Changed makepup line 336 to include CHANGED-ONLY parameter and re-ran my testbed for the third time (far from sure that's the correct choice, but gave it a try anyway).

Code: Select all

else # no user input required
 merge2out_mod # or could implement all this using tcl expect package (and autoexpect) instead, but big install...
 ./merge2out_wiak
 sleep 1 # maybe remove, or adjust duration
 cd ../woof-out_*
 cp -af DISTRO_PKGS_SPECS-* ../ || { printf "\n\nError: DISTRO_PKGS_SPECS-distro-version is missing.\n\nCheck settings and try again. Exiting...\n"; exit 1; }
 # DISTRO_PKGS_SPECS-${DISTRO_BINARY_COMPAT}-${DISTRO_COMPAT_VERSION} is ready for reading at this stage
 # So use -p/--pause flag here to pause script here to allow manual editing of makepup_extra.conf extra packages list.
 pause4adding_packages
 0setup_mod
 1download_mod
 2createpackages_mod CHANGED_ONLY
 3builddistro-Z_mod
and took 47 mins to run through which includes building the devx.sfs. Started building the mksquashfs after 30 minutes, which took 8 mins to run through (a lighter choice of compression could create a sfs in less time, but would be larger).

So no noticeable time difference from my second run

wiak
Posts: 2040
Joined: Tue 11 Dec 2007, 05:12
Location: not Bulgaria

#64 Post by wiak »

rufwoof wrote:Changed makepup line 336 to include CHANGED-ONLY parameter and re-ran my testbed for the third time (far from sure that's the correct choice, but gave it a try anyway).

Code: Select all

else # no user input required
 merge2out_mod # or could implement all this using tcl expect package (and autoexpect) instead, but big install...
 ./merge2out_wiak
 sleep 1 # maybe remove, or adjust duration
 cd ../woof-out_*
 cp -af DISTRO_PKGS_SPECS-* ../ || { printf "\n\nError: DISTRO_PKGS_SPECS-distro-version is missing.\n\nCheck settings and try again. Exiting...\n"; exit 1; }
 # DISTRO_PKGS_SPECS-${DISTRO_BINARY_COMPAT}-${DISTRO_COMPAT_VERSION} is ready for reading at this stage
 # So use -p/--pause flag here to pause script here to allow manual editing of makepup_extra.conf extra packages list.
 pause4adding_packages
 0setup_mod
 1download_mod
 2createpackages_mod CHANGED_ONLY
 3builddistro-Z_mod
)he devx.sfs. Started building the mksquashfs after 30 minutes, which took 8 mins to run through (a lighter choice of compression could create a sfs in less time, but would be larger).

So no noticeable time difference from my second run
No, it won't work that way EDIT: 2createpackages_mod is a function that simply modifies woof-CE original 2createpackages into 2createpackages_wiak. The one you should put CHANGED_ONLY on as option is thus "2createpackages_wiak", which gets called up in makepup. I'll post modified version of makepup that includes --update switch shortly, which does work

If you want to just test more quickly, try changing the makepup line 345 or thereabouts:

Code: Select all

./2createpackages_wiak CHANGED_ONLY
I think that should probably work (maybe...), but as I say I'll probably want to keep non-CHANGED_ONLY behaviour too so 'maybe' planning a new commandline --update/-u switch (or perhaps, more likely maybe, a switch to force change ALL_PACKAGES).

wiak
Last edited by wiak on Thu 31 Aug 2017, 22:20, edited 1 time in total.

User avatar
rufwoof
Posts: 3690
Joined: Mon 24 Feb 2014, 17:47

#65 Post by rufwoof »

Thanks wiak.

I've just made that edit and also set devx creation to false and fired off another run. Unlike prior runs however this one was started with the woof-out_x86_64_x86_64_ubuntu_xenial64 folder deleted.

Personally I'm happy with things as-is (single click <1 hour to build a version updated for all of the latest changes/security patches) :) Takes Puppy into a new dimension.

Post Reply