(OLD) (ARCHIVED) Puppy Linux Discussion Forum Forum Index (OLD) (ARCHIVED) Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info

This forum can also be accessed as http://oldforum.puppylinux.com
It is now read-only and serves only as archives.

Please register over the NEW forum
https://forum.puppylinux.com
and continue your work there. Thank you.

 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups    
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 03 Dec 2020, 19:15
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Auto-build a Puppy iso; single script with optional gui
Moderators: Flash, JohnMurga
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies. View previous topic :: View next topic
Page 5 of 22 [325 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7, ..., 20, 21, 22 Next
Author Message
Keef


Joined: 20 Dec 2007
Posts: 1001
Location: Staffordshire

PostPosted: Thu 31 Aug 2017, 14:43    Post subject:  

wiak

Code:
2createpackages CHANGED_ONLY


I think you maybe right about this. Some time ago there was a debate about this subject, and I found the same option myself. It is not documented (or wasn't then), and got no reaction when I posted about it, despite the more convoluted methods people were using to avoid rebuilding packages.
Anyway, it did work for me. Not tried your script yet, but will give it a try out soon. Internet very unreliable at the moment so downloading is a bit of a pain.

[EDIT] Yes that was the option - found my post buried in the Slacko Woofers thread,
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Thu 31 Aug 2017, 16:47    Post subject:  

Keef wrote:
wiak

Code:
2createpackages CHANGED_ONLY


...

[EDIT] Yes that was the option - found my post buried in the Slacko Woofers thread,


Thanks Keef, I wish I had noticed your post on Slacko woof threads (surprised no one commented on your post, not even 01micko, or picked the tip up - would have been better that it was fully confirmed/documented at that stage since would have saved others this later fluffing-about time...), but I saw the mention of CHANGED_OPTION in 2createpackages and wondered about it but wasn't entirely sure of its purpose till I read through the code - and wanted to do that anyway (or get confirmation from jlst or someone who likely knows) before I could risk modifying makepup on the basis. Anyway, as I say, the run_findpkg routine, EDIT: which is in the _00func script uses support/findpkgs script to check md5sums and so on, and testing suggests it works. Thanks for your confirmation and double-check.

I'm wondering now what other options may be buried in the code or if they are documented outside the code somewhere? Its possible some code options have come from BarryK, and not been noticed as yet (seems unlikely), though the name of that option did suggest the possible purpose.

EDIT: have now found you old post. I see you suggest it has been mentioned somewhere before, though I haven't come across earlier references as yet:

http://www.murga-linux.com/puppy/viewtopic.php?p=870014#870014

It certainly was a bit obscure because tricky reading the code to find its purpose since relies on _00func interaction, involving DISTRO_PKGS_SPECS that is quite involved. Very useful piece of code though.

EDIT2: I've now found another relevant link via google puppy searching (shows what a long history option CHANGED_ONLY actually has...):

http://murga-linux.com/puppy/viewtopic.php?p=368841#368841

EDIT3: I've now finally gotten round to looking at script woof-gui and this is mentioned:

Code:
      <hbox>
        <text use-markup=\"true\"><label>\"<b>This tab is for creating the cut-down packages needed to build a Puppy live-CD</b>. This tab is a frontend for script '2createpackages'\"</label></text>
        <text><label>You must have previously downloaded all the packages, see 'Download pkgs' tab. Now, they need to be processed into a cut-down and Puppy-compatible format. The resulting packages are created in directory 'packages-${DISTRO_FILE_PREFIX}'</label></text>
      </hbox>
      <hbox>
        <text><label>'ALL_PACKAGES' means all chosen packages (see 'Choose pkgs' tab) are processed, directory 'packages-${DISTRO_FILE_PREFIX}' is wiped and all packages re-created. 'CHANGED_ONLY' will automatically process only packages that have changed or added since last time 2createpackages was run. Or, if only one package has been added or modified in the 'Choose pkgs' tab, then it can be explicitly selected here to process just that one:</label></text>
        <combobox><variable>CHOICE_CREATEPACKAGES</variable>${ITEMS_CG}</combobox>
      </hbox>


though I haven't checked as yet where CHANGES_ONLY option is used by the underlying code there.

EDIT4: Ah, here is where woof-gui optionally uses that CHANGES_ONLY option:

Code:
  <button><label>CREATE PACKAGES</label><action>rxvt -title \"Creating packages\" -bg yellow -e ./2createpackages \$CHOICE_CREATEPACKAGES</action><action>xmessage -bg green -center \"Finished. Click 'REPORT' button for a summary.\"</action></button>


wiak

Last edited by wiak on Thu 31 Aug 2017, 18:24; edited 4 times in total
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3725

PostPosted: Thu 31 Aug 2017, 17:21    Post subject:  

Changed makepup line 336 to include CHANGED-ONLY parameter and re-ran my testbed for the third time (far from sure that's the correct choice, but gave it a try anyway).

Code:
else # no user input required
 merge2out_mod # or could implement all this using tcl expect package (and autoexpect) instead, but big install...
 ./merge2out_wiak
 sleep 1 # maybe remove, or adjust duration
 cd ../woof-out_*
 cp -af DISTRO_PKGS_SPECS-* ../ || { printf "\n\nError: DISTRO_PKGS_SPECS-distro-version is missing.\n\nCheck settings and try again. Exiting...\n"; exit 1; }
 # DISTRO_PKGS_SPECS-${DISTRO_BINARY_COMPAT}-${DISTRO_COMPAT_VERSION} is ready for reading at this stage
 # So use -p/--pause flag here to pause script here to allow manual editing of makepup_extra.conf extra packages list.
 pause4adding_packages
 0setup_mod
 1download_mod
 2createpackages_mod CHANGED_ONLY
 3builddistro-Z_mod

and took 47 mins to run through which includes building the devx.sfs. Started building the mksquashfs after 30 minutes, which took 8 mins to run through (a lighter choice of compression could create a sfs in less time, but would be larger).

So no noticeable time difference from my second run
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Thu 31 Aug 2017, 17:28    Post subject:  

rufwoof wrote:
Changed makepup line 336 to include CHANGED-ONLY parameter and re-ran my testbed for the third time (far from sure that's the correct choice, but gave it a try anyway).

Code:
else # no user input required
 merge2out_mod # or could implement all this using tcl expect package (and autoexpect) instead, but big install...
 ./merge2out_wiak
 sleep 1 # maybe remove, or adjust duration
 cd ../woof-out_*
 cp -af DISTRO_PKGS_SPECS-* ../ || { printf "\n\nError: DISTRO_PKGS_SPECS-distro-version is missing.\n\nCheck settings and try again. Exiting...\n"; exit 1; }
 # DISTRO_PKGS_SPECS-${DISTRO_BINARY_COMPAT}-${DISTRO_COMPAT_VERSION} is ready for reading at this stage
 # So use -p/--pause flag here to pause script here to allow manual editing of makepup_extra.conf extra packages list.
 pause4adding_packages
 0setup_mod
 1download_mod
 2createpackages_mod CHANGED_ONLY
 3builddistro-Z_mod
)he devx.sfs. Started building the mksquashfs after 30 minutes, which took 8 mins to run through (a lighter choice of compression could create a sfs in less time, but would be larger).

So no noticeable time difference from my second run


No, it won't work that way EDIT: 2createpackages_mod is a function that simply modifies woof-CE original 2createpackages into 2createpackages_wiak. The one you should put CHANGED_ONLY on as option is thus "2createpackages_wiak", which gets called up in makepup. I'll post modified version of makepup that includes --update switch shortly, which does work

If you want to just test more quickly, try changing the makepup line 345 or thereabouts:

Code:
./2createpackages_wiak CHANGED_ONLY


I think that should probably work (maybe...), but as I say I'll probably want to keep non-CHANGED_ONLY behaviour too so 'maybe' planning a new commandline --update/-u switch (or perhaps, more likely maybe, a switch to force change ALL_PACKAGES).

wiak

Last edited by wiak on Thu 31 Aug 2017, 18:20; edited 1 time in total
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3725

PostPosted: Thu 31 Aug 2017, 18:12    Post subject:  

Thanks wiak.

I've just made that edit and also set devx creation to false and fired off another run. Unlike prior runs however this one was started with the woof-out_x86_64_x86_64_ubuntu_xenial64 folder deleted.

Personally I'm happy with things as-is (single click <1 hour to build a version updated for all of the latest changes/security patches) Smile Takes Puppy into a new dimension.
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3725

PostPosted: Thu 31 Aug 2017, 18:20    Post subject:  

Around 10 mins in and its flying through. Done the 1st and 2nd passes and now just running through the 'Processing ...' where it looks by eye that all of the .DEB files are being seen as unchanged (not being processed), but where all .pet's seem to be being processed (as though they've changed). Again I'm in the dark here so might just be my misinterpretation.
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Thu 31 Aug 2017, 18:29    Post subject:  

peebee wrote:
wiak wrote:
Is there a way provided by 1download to indicate which packages have been updated such that I can automate 2createpackages to selectively operate on just these packages in terms of update?

LOL - I've got a tweak to 1download that I will submit to woof-ce tomorrow for precisely this....


Hi Peebee,

Yes, fair enough to add code to woof-CE 1download script in order to also get a download_list though. I did also notice md5sums don't seem to actually be stored anywhere for non-pets, so a download_list could also be speedier than any existing methods (assuming CHANGED_ONLY method actually works as of now.

wiak
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Thu 31 Aug 2017, 18:32    Post subject:  

rufwoof wrote:
Around 10 mins in and its flying through. Done the 1st and 2nd passes and now just running through the 'Processing ...' where it looks by eye that all of the .DEB files are being seen as unchanged (not being processed), but where all .pet's seem to be being processed (as though they've changed). Again I'm in the dark here so might just be my misinterpretation.


Thanks rufwoof,

Looking forward to your further update on the final result. I'm also a bit in the dark and hence waiting to release new makepup based on any of this; the key all lies in the woof-CE support/findpkgs script, which I haven't worked out its full operation as yet so don't know if CHANGED_ONLY will do as we wish as things stand (may or may not...). support/findpkgs script is itself around twice the size of makepup...!

Peebee's 1download mod to woof-CE itself might prove to be a better method in the end.

support/findpkgs actually seems to be checking out modification dates (I'm not sure) rather than just md5sums (actually, on re-reading the script, it seems to use md5sums instead of actual modification dates) but I haven't worked out the details and the mechanism may not be very efficient as done:

Code:
## DS_DATE and DPS_MODIFY_DATE are md5 checksums
## when these are different it means an update has to happen


wiak
Back to top
View user's profile Send private message 
rufwoof


Joined: 24 Feb 2014
Posts: 3725

PostPosted: Thu 31 Aug 2017, 19:03    Post subject:  

Not sure what happened with that last run of mine. I had set build devx off ... but it still built one. Noticed that I have two CPU temperatures in the panel now so I think running multiple times in a single session (sandboxes ...etc.), might all be getting tangled up in some way.

I'll clean up that testbed and start afresh from a clean reboot and just makepup alone ... but another day (not tonight).
Back to top
View user's profile Send private message 
davids45


Joined: 26 Nov 2006
Posts: 1327
Location: Chatswood, NSW

PostPosted: Fri 01 Sep 2017, 01:34    Post subject: Success!
Subject description: Makepup isos (2) good - now only have pre-existing Woof problems
 

G'day wiak,

Thanks for your previous responses to my problem getting makepup working fully.

I don't know what was going wrong with my initial make-pups but by moving/copying the make-pup directory from my Frugal Slacko-6.9.9.9 root directory to my data partition and running the script in that from a Full Xenial Pup, I achieved an iso of the default slacko Very Happy .

Next evening, hubris to the fore, I made a Xenial Pup, again without problem Laughing . Both took order-of-magnitude 1 hour.

Today, I ran both isos (via manual-install copying of their boot directory files) as new Frugals and both were fine.

Just the common problems in woof-CEs due to missing libs (e.g. libpng.so.12 in slacko) and incompatible/buggy (?) ghostscript files so printing doesn't work - I'll try the "use-the-old gs" files fix later. Scanning is also a variable problem - some packages work, another previews OK but trying an actual scan, it tells me I "don't have enough disk space" which is a bug in that particular package.

When I find the time, I'll see if I can fix these woof-CE issues I have by tinkering with the makepup files.

Thanks again for makepup.

David S.
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Fri 01 Sep 2017, 03:01    Post subject:  

Hello David,

Yes, I was going to ask if you were trying to build on your Puppy save file rather than somewhere on /mnt/home. Yes, that would explain the issues you were having - your builds were running out of resource space and hanging....

Good to hear all good now.

wiak
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Fri 01 Sep 2017, 03:07    Post subject:  

Hi all,

I have just completed a test of using CHANGES_ONLY on the second run of makepup and all went well and the second build (since it had nothing to download or rebuild (aside from some minor config matters and the squashfs itself) successfully completed the update in the order of 15min...

My first run was in an empty folder using default version 0.0.2:

Code:
./makepup


That successfully build the default Slacko pup as normal.

I then modified makepup by changing line 348 from:

Code:
./2createpackages_wiak


to

Code:
./2createpackages_wiak CHANGED_ONLY


(leaving all the previous build directories untouched)

and that just zipped through most everything and the 'less than 15 minutes to create' update/rebuild seems to be fine.

More testing required, but this CHANGED_ONLY method seems to be fine (though can't be sure since no packages have needed updated as yet, so can't test if it will work that I can think of). Oh, I know, I'll remove one of the packages from my local-repositories and run that modified makepup again to see if it picks that up and re-installs the missing package...

I'll report back in half an hour or whatever it all takes.

wiak

wiak
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Fri 01 Sep 2017, 03:38    Post subject:  

Okay, I removed the previously downloaded files concerned with busybox and also those concerning autologin from local-repositories/x86/packages-pet. I also removed bash and coreutils from local-repositories/x86/packages-tgz_txz-14.2. And then ran the makepup containing CHANGED_ONLY line again.

It successfully noticed the missing files and quickly replaced them and finished the overall rebuild once again in under 15 minutes.

I'll do one more test for now. This time I'll damage a couple of files in these local-repository folders (change a few bytes with a hex editor after first manually checking their md5sums). Then I'll see if running CHANGED_ONLY makepup successfully notices that and fixes everything in that hoped for less than 15 min again. Reporting back soon.

EDIT: Actually I used a combination of /dev/random and dd to change one byte in both busybox....pet and in bash...txz. I could of course simply have used a different file as a replacement for busybox (renaming to busybox etc, which would also guarantee a different md5sum from the original of course... same with bash and any other program I wanted to 'corrupt').

EDIT2: So the answer to the above is NO... - re-running makepup with CHANGES_ONLY mod did not fix either the md5sum corrupted busybox dotpet nor the bash txz. So my understanding of how it might work isn't correct. However, might still be fine if makepup with CHANGES_ONLY addition is able to update when the package has a different version number (?). I'll try now modifying the busybox dotpet and bash txz package names with different version numbers than their originals and see if they get auto-replaced. Otherwise I can't think of a way of testing if auto-updating with CHANGES_ONLY usefully works... (well, at least we know now it DOES fix missing local-repository packages, but not md5sum corrupted ones... maybe it does somehow check file modification dates?) until such time as an actual package is known to have been updated up-stream. Again, I'll report back soon.

EDIT3: I changed the version numbers of the local-repository stored files for autologin (pet), busybox (pet), bash (txz) and coreutils (txz) and on running makepup with CHANGED_ONLY mod the correct current versions were downloaded (the changed ones weren't deleted from inside local-repositories). and then the build proceeded successfully (total build time again less than 15 minutes on my Intel Core II duo, 2 GHz CPU, 2 GB quite old laptop and very slow broadband of less than 500 kB/sec). According to file woof-installed-packages, everything now seems to be fine again. If that is so then CHANGED_ONLY method seems to work, though I have no idea how it actually determines what needs updated... More testing required for better certainty, but looking good...

wiak
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Fri 01 Sep 2017, 05:06    Post subject:  

OK, my opinion is that CHANGES_ONLY update method works fine, and fast.

I'll implement that in next makepup in some manner or other. Probably upload version 0.0.3 with that change sometime next day or two.

If that method ends up not working so well as it seems to (or I hope) I can always use alternative method (based on list of new 1download packages) in later makepup version.

wiak
Back to top
View user's profile Send private message 
wiak

Joined: 11 Dec 2007
Posts: 2075
Location: not Bulgaria

PostPosted: Fri 01 Sep 2017, 06:06    Post subject:  

Tip:

If you have tons of RAM on your system, try putting a copy of the makepup script in /tmp and build your Puppy in there. Should be much faster than building to a harddrive. Having said that, this tip will not improve bottleneck of download speed (the issue I have with my rural broadband connection).

Got the idea behind that tip from this nice woof building tutorial by Iguleder, which I came across only today:

http://www.murga-linux.com/puppy/viewtopic.php?t=47331&ppage=2000

However, I don't have much RAM (sad) so I can't test the tip.

Iguleder's tutorial is very old by the way (read as 'ancient'), so read it more generally from that out-of-date point of view.

More up-to-date info in thread below, but not much technical detail:

http://puppylinux.com/woof-ce.html

And remember: makepup requires no knowledge of git, github and DOES NOT require git to be installed on your puppy system (and DOES NOT need a devx loaded unless you need binaries stripped) - just needs your standard, relatively recent Pup for running makepup script on... I'm hoping it provides a relatively simple introduction to building a woof-CE Puppy...

I don't know if there is much documentation of the internal technical aspects of woof-CE operation actually. Wish I did rather than just reading the scripts in there!

Fortunately, you don't need to know any of this technical stuff to build a woof-CE puppy using makepup script. Just use the script and that happens automatically for you...

I'm planning a short rest now before proceeding with version 0.0.3 of makepup.

wiak

Last edited by wiak on Tue 05 Sep 2017, 19:20; edited 2 times in total
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 5 of 22 [325 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7, ..., 20, 21, 22 Next
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies. View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0931s ][ Queries: 12 (0.0153s) ][ GZIP on ]