PPkg - script-based packager

Under development: PCMCIA, wireless, etc.
Post Reply
Message
Author
noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

PPkg - script-based packager

#1 Post by noryb009 »

PPkg is a project I've been working on that is a script based packager. It takes a "PUPPYPKG" file, reads the data in it, and compiles a puppy package from it.

PPkg has many improvements over new2dir and dir2pet, including:
- Making packages without answering any questions.
- Exporting in PET, SFS and/or RXZ formats.
- Automatic package splitting.
- Package signing implemented (but it can't be used with the PET format)
- Being able to easily share your packages and how you made them (most "pBuild" files are less then 1KB)
- Able to update a package by changing as little as one line (in some cases)
- With PPkg, you can compile an entire repository of packages for any architecture and for any puplet, in one command

The problem with PPkg is that there are only a few build scripts. If you have any experience with packaging, any help would be appreciated.
Last edited by noryb009 on Sat 18 Feb 2012, 20:32, edited 1 time in total.

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#2 Post by Lobster »

Many thanks :)
"PUPPYPKG" file
What is that? Is this a development of PET?
PPkg has many improvements over new2dir and dir2pet, including:
Does any part of the program exist?
Can you give an example of usage? :)
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

User avatar
harii4
Posts: 448
Joined: Fri 30 Jan 2009, 04:08
Location: La Porte City, IA , U.S.A.
Contact:

#3 Post by harii4 »

you would need puppy build scripts for each Puppy series (2-5) right?
sounds cool :)
Last edited by harii4 on Mon 13 Feb 2012, 11:45, edited 1 time in total.
3.01 Fat Free / Fire Hydrant featherweight/ TXZ_pup / 431JP2012
----------------------------------------------------------------------------------------
Peace and Justice are two sides of the same coin.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#4 Post by noryb009 »

What is that? Is this a development of PET?
A PUPPYPKG file is a file that has info about a package. A template can be found here.
Does any part of the program exist?
Can you give an example of usage?
It does exist. Right now, there are no packages in the script repository, but some sample commands are:

Code: Select all

ppkg --pet # make a PET package from the PUPPYPKG in the current folder
ppkg --pet --sfs --rxz # make a PET, SFS and RXZ package from the PUPPYPKG in the current folder
ppkg -P # compress the PUPPYPKG and any local sources (like patches) to a .pbuild.tar.gz file for easy sharing
ppkg --sfs -p packagename.pbuild.tar.gz # make a SFS from a pBuild
ppkg --pet -p *.pbuild.tar.gz # make a PET for all the pbuilds in the current folder
git clone git://github.com/noryb009/ppkg-scripts.git; ppkg --pet -P core/* # download the ppkg-scripts repository and make a pet package for each core package - this won't work until there are some packages in the repository
As you can see, ppkg is a tool that makes one (or two) commands save a LOT of time.
you would need puppy build scripts for each Puppy series (2-5) right?
Not exactly. While there is a lot different between the series, the package management is mostly the same. It's possible to include some series/puplet specific code in a PUPPYPKG file, but most packages won't need it.

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#5 Post by Lobster »

there are no packages

:?

Will you be providing some?
Or is this an early idea you are developing and intend to provide packages for?

I am sorry I do not understand what you are doing and why? :oops:

Perhaps someone can dumb it down for me? 8)

Many thanks :)
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#6 Post by noryb009 »

Will you be providing some?
Or is this an early idea you are developing and intend to provide packages for?
There will be a repository of PUPPYPKG scripts, but I haven't been able to write enough for a base system yet.
I am sorry I do not understand what you are doing and why?
Currently, to package a program, you have to run:

Code: Select all

wget http://path.com/to/file-1.0.tar.gz
tar xzf file-1.0.tar.gz
cd file-1.0
*find needed patches and patch the source*
./configure
make
new2dir make install
*answer a bunch of questions*
dir2pet ../test-1.0-i686
*answer more questions*
*test pet*
The problem with this is that if another user wants to create a PET of the same package for a different architecture or different puplet, or if they want to update the current PET, they have to do ALL the steps again.

PPkg helps the original packagers skip a few steps, and the other users skip all the steps. Packagers just need to fill out a form about a package, and edit a few lines. While this is about the same amount of work as compiling it normally, it makes compiling the package again very easy. They only have to run:

Code: Select all

ppkg --pet -p test-1.0.pbuild.tar.gz

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#7 Post by technosaurus »

see src2pkg for package builds and pcompile for the GUI-
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#8 Post by Lobster »

While this is about the same amount of work as compiling it normally, it makes compiling the package again very easy.
Sorry for being so obtuse.
:idea: I gets it :idea:

Eh Ma Ho
How Wonderful (Tibetan)

So once the template is provided
a package can be compiled with the devx for 64 bit, Lucid, Slacko, Saluki, Racy, Wary
ARM and MIPS (when available) etc?

We Have Lift Off (Nasa) :)
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#9 Post by noryb009 »

So once the template is provided a package can be compiled with the devx for 64 bit, Lucid, Slacko, Saluki, Racy, Wary ARM and MIPS (when available) etc?
Yes, once a packager creates a PUPPYPKG file for a program, it can be used for any puplet or architecture (pretty much).
see src2pkg for package builds and pcompile for the GUI-
I haven't personally used src2pkg, but from what I have read, it is a good automatic packager for personal use. It can build a packages quickly, with hardly any setup or knowledge. However, I don't think it is powerful enough to be a distribution's package building system because of this line (line 117 in 15-make_package_pet):
DATABASE_ENTRY="${NAME}${TAG}-${VERSION}|<snip>|${NAME}${TAG}-${VERSION}-${BUILD}.pet|unknown|${SUMMARY}|<snip>"
As you can see the field that should have the dependencies has a hard coded "unknown" in it. That isn't good, especially if EVERY package doesn't have dependencies defined.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#10 Post by noryb009 »

Attached is a compiled version of the latest commit (the PET file).

Also attached is a pbuild.tar.gz file. Once you have installed ppkg, you can use the pbuild file to create a new pet package. All you have to do is download the attached pbuild (to a different folder then the PET to avoid overwriting it), then run the following command:

Code: Select all

ppkg --pet -p ppkg-git-2012.02.13-1.pbuild.tar.gz
Please note: you need the devx, and if you are not in the same folder as the pbuild, you must specify a path to it.

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#11 Post by sc0ttman »

This looks good.. Have you seen Puppizard, by Iguleder?

http://www.murga-linux.com/puppy/viewtopic.php?t=69248
You might be able to steal some of his build scripts... dunno...
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#12 Post by technosaurus »

deleted stray post
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#13 Post by amigo »

src2pkg performs literally thousands of steps when configuring, compiling and packaging, so I don't think one line (line 117 in 15-make_package_pet) should be a show-stopper. It's simply a place-holder action which is waiting for a more complete routine to make it more useful. I decided to not waste time on improving that because the pet package format is so flawed and fluid.

As far as I can tell, pet packages and Puppy do not contain enough information to accurately determine the requirements of a package without resorting to very round-about methods. Anyone with any ideas about how to accomplish this are welcome to submit patches to src2pkg to fix this. We've been talking about this for a long time now, but I don't see where anyone has grasped the essential ideas necessary to make this work for puppy -it simply is inadequate to say: well, file belongs to 'base' or general-libs.sfs.

Still, if you have a list of what belongs to each unit, whether that is an sfs, some base installation and the contents of individual packages, then a system can be devised.

Finally, if you study the code there a bit, you'll see that it will use an already-prepared pet.specs file if one is available, instead of trying to generate one. So, the tedious way to overcome this is to prepare such a file for each package. You can always run the build once and then copy the resulting specs file into the CWD and edit it to complete it, then re-run the build to have it included. You could also use the src2pkg 'extensions' feature and write abit of code for an extension which pops a fancy gui and helps the user input the necessary data. Gettiung that data is the hard part -show me where there is enough info to automatically fill in thse fields and I'll work on it. But really, pets and all package management in puppy is woefully inadeqate for the job.

I would hope that you would do a bit of work to make src2pkg do a better job with this, rather than re-inventing what is a very good wheel. You have to understand that src2pkg has been under development and *heavy* use for nearly nine years now -there are thousands of pitfalls in config/compile/package processes which you will not find for years to come. And still, the pet format as it is not up to handling what you really want...

I have a system here where every single package is created using src2pkg, so I well know it is possible. But, the package creator has to be closely co-ordinated with the package format to achieve superior results. I wanted the capability to do dependency-resolution, so I designed my own package format and routines within src2pkg to go with that so that it is possible. I could certainly build such routines into src2pkg for other package formats, but my time is valuable and I would not waste it on what is essentially a no-go package format. src2pkg can only produce accurate dependency information for each package when all the packages or the package manager itself provide this info.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#14 Post by noryb009 »

I decided to not waste time on improving that because the pet package format is so flawed and fluid.
I completely agree.
I would hope that you would do a bit of work to make src2pkg do a better job with this, rather than re-inventing what is a very good wheel. You have to understand that src2pkg has been under development and *heavy* use for nearly nine years now -there are thousands of pitfalls in config/compile/package processes which you will not find for years to come.
Ppkg and src2pkg are similar, but work in different ways. Ppkg is meant to compile packages from pre-built scripts, allowing advanced packages to be created multiple times. Src2pkg is meant to compile packages, allowing packages to be created quickly. It also has a build script format, but from what I've seen, it does not contain much info about the package.

However, ppkg was not written for making PET packages. It was built for making PET2 packages - whatever they may be. Those packages would need some info that cannot be found in the source package, like dependency packages (different from dependency files), if a package cannot be installed with another package, and more, which is why PUPPYPKG files are used.

I haven't tried it, but it should be possible to use src2pkg to compile a package in a PUPPYPKG script, while letting Ppkg handle the package specs, signing, etc.
And still, the pet format as it is not up to handling what you really want...
See the top post here

EDIT: I have uploaded a few scripts to the online repository. To compile one, just download and extract the repository, then run:

Code: Select all

cd core
ppkg --pet -p bash # replace "bash" with the program(s) you want to compile

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#15 Post by amigo »

"does not contain much info about the package" All the info is *generated* by src2pkg at package-creation time. That means it is always current, uniformly-formatted and correct -and that you don't have to supply it all. Early on in the development-cycle of src2pkg, I could already see that I usually spent more time creating/editing the package description file than on anything else. So, one of the first automated-to-semi-automated thngs src2pkg did was generate them whenever possible, or produce a workable template at least.

In a couple of day I'm gonna post some example files for the tpkg-format (as produced by src2pkg) and output from some of the tipical commands that are available with the tpkg tools.

The src2pkg build scripts are designed to need as little manual modifications as possible. It's tedious to always have to be creating or editing them. They contain only essential information -transient or semipermanent options which can apply to any build are picked up from the conf file or the src2pkg command-line options. The minimum code for a build script is just *3* lines, and don't forget that sometimes you don't need one at all (a three-liner can't do any more than you can without any script at all). But, even a three-liner with a suffix like '.auto' gives you a visual clue that this package needs *no special treatment*, like package-dependent configure options, extra sources or the removal of files. You can even automatically upgrade a src2pkg script when you update the source version with just a command-line switch -I use thus heavily for building X & Co., where there are hundreds of source archives -just repace with the new tarballs and run: src2pkg -U -X name-of-tarball (the tarball name is usually handled with wildcards) -the script is updated (SOURCE_NAME and VERSION) automatically and then run...

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#16 Post by noryb009 »

"does not contain much info about the package"
All the info is *generated* by src2pkg at package-creation time. That means it is always current, uniformly-formatted and correct -and that you don't have to supply it all.
While the basic info needed for the PET format is generated (name, version, description), it doesn't (and can't) get some info, like it's dependency packages (you can automate some, but can't catch everything), if a package should be split (not into a DEV package, but one source into 2 different packages like "linux" and "linux-source"), and what architectures it can be built for.

The ppkg build scripts are designed to need manual modifications, but automate generic parts. It's tedious to always have to be generating them then merging the old version's options. They contain only essential information and info that removes the need for command-line options -transient or semipermanent options which can apply to any build are picked up from the conf file. The minimum code for a build script is just *5* lines, 1 of which is "}", and don't forget that sometimes you just need to fill in the source file, name, and version. You can even automatically download and compile a new version when you update one line. Where there are hundreds of source archives - just run `ppkg --pet -p *.pbuild.tar.gz` to compile everything.
See what I did there? :P

Ppkg doesn't generate anything, it gives you a template to start off with. If Ppkg was made for the PET format, packages would only have to fill in 8 fields - name, version, revision, description, category, arch, deps, and sources. In comparison, src2pkg needs 1.5 (sources and partially description), ignores 2 (arch and deps), and generates the rest (which only takes a minute to fill in to a PUPPYPKG file).
A PUPPYPKG file is easy to write, but allows packagers the power needed for "advanced" packages which need lots of patches, make flags, custom configuration files, symlinks, and moving files between split packages.
A big part of Ppkg is pBuild files - tarballs that can be compiled, which include any local sources that cannot be downloaded from the internet, without needing your own server to host the files.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#17 Post by amigo »

"get some info, like it's dependency packages" You can only get this info if it available. I mean only by having a complete list of every file and link contained in or created by a package -for every package. This is where the package format and the package-management system comes into play. If the installation of the packafe doesn't preserve or generate such a list, then dependency reolution is impossible. I'll give you an example.

Slackware packages do not contain a file/link list. But, on installation a file list is created and appended to an info file generated during installation. The links can be derived from the doinst.sh script by expensive parsing. Re-moving a Slackware package actually does a form of dependency resolving, in that before a file is removed from the system,all the files lists (the ones generated at installation time) are checked to see if the file also belongs to some other package. If so, then it is not removed. This is the only way to sanel remove or upgrade a package.

The reason that no dependency-resolution is possible with Slackware - vis-a-vis installing other packages which are needed by the one you want to install, is that when slackware packages are built, no information about these other packages is included. In order to generate that info you need the file-lists from all packages. It is a chicken-egg situation, but works fine if you build the whole system that way. And the file/link list for each package is the only thing needed to accomplish it. for instance, when you have the content created for a new package, you run or 'ldd' or some better program on each ELF object in the content. That will show you the *library* dependencies the object has. For each needed library, you look it up in the database to see which package it belongs to. The resulting package name is what your new package depends on. Simple -seemingly. More elaborate and complete dependency information must be found using more exhaustive methods -and sometimes info has to be supplied manually. for instance, there is hardly a way to determine that the 'man' program really needs 'groff' in order to function. Still, src2pkg has been able to overcome some of these snafus and you can always add any special depends info -without giving up on the automatically-generated info -which is dynamic and can *only* be generated at package-creation time. It does not work to use some arbitrary hand-created list of dependencies for packages, because as soon as you change compile-time options or the mix of packages installed at package-creation time, then the dependencies are subject to change.

src2pkg *does* automatically determine whether a package should be split into sub-packages or not -but only if you tell it to use that option. Not everyone wants split packages -Slackware doesn't use them at all, so it has to be a controllable option and not an arbitrary action. Plus, there are times when even you will not want to split a package. Take for instance the package 'man-pages'. That's a huge set of man-pages for system functions, etc. It is comprised of only man-pages. So, you surely wouldn't want to split the docs package there -nothing would be left in the original package. src2pkg even figures out which man-pages should be part of the 'devel' package, which should be part of the 'nls' package and which are english-only man-pages for a normal user -not a developer.

Just for the record, about how many packages and sfs's have you created in total? Have you even partially implemented your new package format with tools that create the packages and tools which install/upgrade/remove them? Have you studied any of the other package formats and their features?

I have created *way* over 100,000 packages using src2pkg -using other peoples package formats -including *.pets. I haven't gotten there yet with my new package format, but it is certainly over 5,000 -my system currently has about 1500 packages and they have been mass-rebuilt many times already.

noryb009
Posts: 634
Joined: Sat 20 Mar 2010, 22:28

#18 Post by noryb009 »

If the installation of the packafe doesn't preserve or generate such a list, then dependency reolution is impossible.
It isn't impossible if the user inputs dependencies - which is more accurate and flexable.
Slackware packages do not contain a file/link list. But, on installation a file list is created
Same as puppy. However, this means that you have to have every dependency installed before you build a package.
It does not work to use some arbitrary hand-created list of dependencies for packages, because as soon as you change compile-time options or the mix of packages installed at package-creation time, then the dependencies are subject to change.
This was one of the only things left to do in Ppkg - package-time library dependencies. I haven't done it until now because it wouldn't be used with the current package formats (and it's harder to bugtest then just signing). I'm planning to add needed libraries to the dependency array, and provided libraries to the non-existent provided array.
Just for the record, about how many packages and sfs's have you created in total? Have you even partially implemented your new package format with tools that create the packages and tools which install/upgrade/remove them? Have you studied any of the other package formats and their features?
I've not created too many PET packages - only about 30 - not using Ppkg. I hated the experience. Compared to any other distribution I've tried, which includes Mint, Arch, and Gentoo, it was like the stone age of packaging. I built Ppkg by taking the strong points of these package systems, and more. Ppkg can use any building system (make, cmake, ...), from any source (a tarball, git, svn, etc.), for any puppy package format (PET, SFS, and RXZ). Obviously, Ppkg is new, so it hasn't built thousands of packages, but the few scripts I have written so far in my spare time has been very easy.

Post Reply