Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 23 Sep 2014, 08:29
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
PPkg - script-based packager
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [18 Posts]   Goto page: 1, 2 Next
Author Message
noryb009

Joined: 20 Mar 2010
Posts: 539

PostPosted: Sun 12 Feb 2012, 21:38    Post subject:  PPkg - script-based packager  

PPkg is a project I've been working on that is a script based packager. It takes a "PUPPYPKG" file, reads the data in it, and compiles a puppy package from it.

PPkg has many improvements over new2dir and dir2pet, including:
- Making packages without answering any questions.
- Exporting in PET, SFS and/or RXZ formats.
- Automatic package splitting.
- Package signing implemented (but it can't be used with the PET format)
- Being able to easily share your packages and how you made them (most "pBuild" files are less then 1KB)
- Able to update a package by changing as little as one line (in some cases)
- With PPkg, you can compile an entire repository of packages for any architecture and for any puplet, in one command

The problem with PPkg is that there are only a few build scripts. If you have any experience with packaging, any help would be appreciated.

Last edited by noryb009 on Sat 18 Feb 2012, 16:32; edited 1 time in total
Back to top
View user's profile Send private message 
Lobster
Official Crustacean


Joined: 04 May 2005
Posts: 15117
Location: Paradox Realm

PostPosted: Sun 12 Feb 2012, 22:39    Post subject:  

Many thanks Smile

Quote:
"PUPPYPKG" file


What is that? Is this a development of PET?

Quote:
PPkg has many improvements over new2dir and dir2pet, including:


Does any part of the program exist?
Can you give an example of usage? Smile

_________________
Puppy WIKI
Back to top
View user's profile Send private message Visit poster's website 
harii4


Joined: 30 Jan 2009
Posts: 444
Location: La Porte City, IA , U.S.A.

PostPosted: Sun 12 Feb 2012, 22:56    Post subject:  

you would need puppy build scripts for each Puppy series (2-5) right?
sounds cool Smile

_________________
3.01 Fat Free / Fire Hydrant featherweight/ TXZ_pup / 431JP2012
----------------------------------------------------------------------------------------
Peace and Justice are two sides of the same coin.

Last edited by harii4 on Mon 13 Feb 2012, 07:45; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
noryb009

Joined: 20 Mar 2010
Posts: 539

PostPosted: Sun 12 Feb 2012, 23:19    Post subject:  

Quote:
What is that? Is this a development of PET?

A PUPPYPKG file is a file that has info about a package. A template can be found here.

Quote:
Does any part of the program exist?
Can you give an example of usage?

It does exist. Right now, there are no packages in the script repository, but some sample commands are:
Code:
ppkg --pet # make a PET package from the PUPPYPKG in the current folder
ppkg --pet --sfs --rxz # make a PET, SFS and RXZ package from the PUPPYPKG in the current folder
ppkg -P # compress the PUPPYPKG and any local sources (like patches) to a .pbuild.tar.gz file for easy sharing
ppkg --sfs -p packagename.pbuild.tar.gz # make a SFS from a pBuild
ppkg --pet -p *.pbuild.tar.gz # make a PET for all the pbuilds in the current folder
git clone git://github.com/noryb009/ppkg-scripts.git; ppkg --pet -P core/* # download the ppkg-scripts repository and make a pet package for each core package - this won't work until there are some packages in the repository

As you can see, ppkg is a tool that makes one (or two) commands save a LOT of time.

Quote:
you would need puppy build scripts for each Puppy series (2-5) right?

Not exactly. While there is a lot different between the series, the package management is mostly the same. It's possible to include some series/puplet specific code in a PUPPYPKG file, but most packages won't need it.
Back to top
View user's profile Send private message 
Lobster
Official Crustacean


Joined: 04 May 2005
Posts: 15117
Location: Paradox Realm

PostPosted: Sun 12 Feb 2012, 23:31    Post subject:  

Quote:
there are no packages


Confused

Will you be providing some?
Or is this an early idea you are developing and intend to provide packages for?

I am sorry I do not understand what you are doing and why? Embarassed

Perhaps someone can dumb it down for me? Cool

Many thanks Smile

_________________
Puppy WIKI
Back to top
View user's profile Send private message Visit poster's website 
noryb009

Joined: 20 Mar 2010
Posts: 539

PostPosted: Mon 13 Feb 2012, 00:13    Post subject:  

Quote:
Will you be providing some?
Or is this an early idea you are developing and intend to provide packages for?

There will be a repository of PUPPYPKG scripts, but I haven't been able to write enough for a base system yet.

Quote:
I am sorry I do not understand what you are doing and why?

Currently, to package a program, you have to run:
Code:
wget http://path.com/to/file-1.0.tar.gz
tar xzf file-1.0.tar.gz
cd file-1.0
*find needed patches and patch the source*
./configure
make
new2dir make install
*answer a bunch of questions*
dir2pet ../test-1.0-i686
*answer more questions*
*test pet*

The problem with this is that if another user wants to create a PET of the same package for a different architecture or different puplet, or if they want to update the current PET, they have to do ALL the steps again.

PPkg helps the original packagers skip a few steps, and the other users skip all the steps. Packagers just need to fill out a form about a package, and edit a few lines. While this is about the same amount of work as compiling it normally, it makes compiling the package again very easy. They only have to run:
Code:
ppkg --pet -p test-1.0.pbuild.tar.gz
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4351

PostPosted: Mon 13 Feb 2012, 02:47    Post subject:  

see src2pkg for package builds and pcompile for the GUI-
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Lobster
Official Crustacean


Joined: 04 May 2005
Posts: 15117
Location: Paradox Realm

PostPosted: Mon 13 Feb 2012, 03:09    Post subject:    

Quote:
While this is about the same amount of work as compiling it normally, it makes compiling the package again very easy.


Sorry for being so obtuse.
Idea I gets it Idea

Eh Ma Ho
How Wonderful (Tibetan)

So once the template is provided
a package can be compiled with the devx for 64 bit, Lucid, Slacko, Saluki, Racy, Wary
ARM and MIPS (when available) etc?

We Have Lift Off (Nasa) Smile

_________________
Puppy WIKI
Back to top
View user's profile Send private message Visit poster's website 
noryb009

Joined: 20 Mar 2010
Posts: 539

PostPosted: Mon 13 Feb 2012, 18:33    Post subject:  

Quote:
So once the template is provided a package can be compiled with the devx for 64 bit, Lucid, Slacko, Saluki, Racy, Wary ARM and MIPS (when available) etc?

Yes, once a packager creates a PUPPYPKG file for a program, it can be used for any puplet or architecture (pretty much).

Quote:
see src2pkg for package builds and pcompile for the GUI-

I haven't personally used src2pkg, but from what I have read, it is a good automatic packager for personal use. It can build a packages quickly, with hardly any setup or knowledge. However, I don't think it is powerful enough to be a distribution's package building system because of this line (line 117 in 15-make_package_pet):
Quote:
DATABASE_ENTRY="${NAME}${TAG}-${VERSION}|<snip>|${NAME}${TAG}-${VERSION}-${BUILD}.pet|unknown|${SUMMARY}|<snip>"

As you can see the field that should have the dependencies has a hard coded "unknown" in it. That isn't good, especially if EVERY package doesn't have dependencies defined.
Back to top
View user's profile Send private message 
noryb009

Joined: 20 Mar 2010
Posts: 539

PostPosted: Mon 13 Feb 2012, 19:31    Post subject:  

Attached is a compiled version of the latest commit (the PET file).

Also attached is a pbuild.tar.gz file. Once you have installed ppkg, you can use the pbuild file to create a new pet package. All you have to do is download the attached pbuild (to a different folder then the PET to avoid overwriting it), then run the following command:
Code:
ppkg --pet -p ppkg-git-2012.02.13-1.pbuild.tar.gz

Please note: you need the devx, and if you are not in the same folder as the pbuild, you must specify a path to it.
ppkg-git-2012.02.13-1-any.pet
Description 
pet

 Download 
Filename  ppkg-git-2012.02.13-1-any.pet 
Filesize  12.36 KB 
Downloaded  232 Time(s) 
ppkg-git-2012.02.13-1.pbuild.tar.gz
Description 
gz

 Download 
Filename  ppkg-git-2012.02.13-1.pbuild.tar.gz 
Filesize  558 Bytes 
Downloaded  235 Time(s) 
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2376
Location: UK

PostPosted: Thu 16 Feb 2012, 18:04    Post subject:  

This looks good.. Have you seen Puppizard, by Iguleder?

http://www.murga-linux.com/puppy/viewtopic.php?t=69248
You might be able to steal some of his build scripts... dunno...

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4351

PostPosted: Fri 17 Feb 2012, 02:06    Post subject:  

deleted stray post
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Fri 17 Feb 2012, 04:10    Post subject:  

src2pkg performs literally thousands of steps when configuring, compiling and packaging, so I don't think one line (line 117 in 15-make_package_pet) should be a show-stopper. It's simply a place-holder action which is waiting for a more complete routine to make it more useful. I decided to not waste time on improving that because the pet package format is so flawed and fluid.

As far as I can tell, pet packages and Puppy do not contain enough information to accurately determine the requirements of a package without resorting to very round-about methods. Anyone with any ideas about how to accomplish this are welcome to submit patches to src2pkg to fix this. We've been talking about this for a long time now, but I don't see where anyone has grasped the essential ideas necessary to make this work for puppy -it simply is inadequate to say: well, file belongs to 'base' or general-libs.sfs.

Still, if you have a list of what belongs to each unit, whether that is an sfs, some base installation and the contents of individual packages, then a system can be devised.

Finally, if you study the code there a bit, you'll see that it will use an already-prepared pet.specs file if one is available, instead of trying to generate one. So, the tedious way to overcome this is to prepare such a file for each package. You can always run the build once and then copy the resulting specs file into the CWD and edit it to complete it, then re-run the build to have it included. You could also use the src2pkg 'extensions' feature and write abit of code for an extension which pops a fancy gui and helps the user input the necessary data. Gettiung that data is the hard part -show me where there is enough info to automatically fill in thse fields and I'll work on it. But really, pets and all package management in puppy is woefully inadeqate for the job.

I would hope that you would do a bit of work to make src2pkg do a better job with this, rather than re-inventing what is a very good wheel. You have to understand that src2pkg has been under development and *heavy* use for nearly nine years now -there are thousands of pitfalls in config/compile/package processes which you will not find for years to come. And still, the pet format as it is not up to handling what you really want...

I have a system here where every single package is created using src2pkg, so I well know it is possible. But, the package creator has to be closely co-ordinated with the package format to achieve superior results. I wanted the capability to do dependency-resolution, so I designed my own package format and routines within src2pkg to go with that so that it is possible. I could certainly build such routines into src2pkg for other package formats, but my time is valuable and I would not waste it on what is essentially a no-go package format. src2pkg can only produce accurate dependency information for each package when all the packages or the package manager itself provide this info.
Back to top
View user's profile Send private message 
noryb009

Joined: 20 Mar 2010
Posts: 539

PostPosted: Sat 18 Feb 2012, 16:17    Post subject:  

Quote:
I decided to not waste time on improving that because the pet package format is so flawed and fluid.

I completely agree.

Quote:
I would hope that you would do a bit of work to make src2pkg do a better job with this, rather than re-inventing what is a very good wheel. You have to understand that src2pkg has been under development and *heavy* use for nearly nine years now -there are thousands of pitfalls in config/compile/package processes which you will not find for years to come.


Ppkg and src2pkg are similar, but work in different ways. Ppkg is meant to compile packages from pre-built scripts, allowing advanced packages to be created multiple times. Src2pkg is meant to compile packages, allowing packages to be created quickly. It also has a build script format, but from what I've seen, it does not contain much info about the package.

However, ppkg was not written for making PET packages. It was built for making PET2 packages - whatever they may be. Those packages would need some info that cannot be found in the source package, like dependency packages (different from dependency files), if a package cannot be installed with another package, and more, which is why PUPPYPKG files are used.

I haven't tried it, but it should be possible to use src2pkg to compile a package in a PUPPYPKG script, while letting Ppkg handle the package specs, signing, etc.

Quote:
And still, the pet format as it is not up to handling what you really want...

See the top post here

EDIT: I have uploaded a few scripts to the online repository. To compile one, just download and extract the repository, then run:
Code:
cd core
ppkg --pet -p bash # replace "bash" with the program(s) you want to compile
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Sat 18 Feb 2012, 18:05    Post subject:  

"does not contain much info about the package" All the info is *generated* by src2pkg at package-creation time. That means it is always current, uniformly-formatted and correct -and that you don't have to supply it all. Early on in the development-cycle of src2pkg, I could already see that I usually spent more time creating/editing the package description file than on anything else. So, one of the first automated-to-semi-automated thngs src2pkg did was generate them whenever possible, or produce a workable template at least.

In a couple of day I'm gonna post some example files for the tpkg-format (as produced by src2pkg) and output from some of the tipical commands that are available with the tpkg tools.

The src2pkg build scripts are designed to need as little manual modifications as possible. It's tedious to always have to be creating or editing them. They contain only essential information -transient or semipermanent options which can apply to any build are picked up from the conf file or the src2pkg command-line options. The minimum code for a build script is just *3* lines, and don't forget that sometimes you don't need one at all (a three-liner can't do any more than you can without any script at all). But, even a three-liner with a suffix like '.auto' gives you a visual clue that this package needs *no special treatment*, like package-dependent configure options, extra sources or the removal of files. You can even automatically upgrade a src2pkg script when you update the source version with just a command-line switch -I use thus heavily for building X & Co., where there are hundreds of source archives -just repace with the new tarballs and run: src2pkg -U -X name-of-tarball (the tarball name is usually handled with wildcards) -the script is updated (SOURCE_NAME and VERSION) automatically and then run...
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [18 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1200s ][ Queries: 13 (0.0110s) ][ GZIP on ]