[RELEASE] src2pkg-3.0 - create packages from source

Miscellaneous tools
Message
Author
amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#21 Post by amigo »

"Notice -march=pentium4 -mtune=i686" I'll work on making it possible to get what you want there. The '--arch=??' option to src2pkg only sets the ARCH portion of the package name and does not directly affect the gcc options.

"Are the files in /usr/include sometimes used for linking/dependicies during build process?" Yes, of course, this is where development headers(include files) are located. They are not used at run-time.

"tar-1.13" So, the error occurs only and always when there is a space in the name? That can be fixed -but seeing this reminds me that for pets a normal version of tar could be used instead.

"build a whole system using 'better' settings" Since nearly everything on the system links to glibc, a small performance improvement can be had by compiling glibc to match your hardware -**but** re-compiling glibc implies re-compiling everything that gets linked to it. Further, very few sources/programs actually use 'enhanced' CPU features -the big exception is audio/video editing software and high-performance number-crunching software.

So, optimizing libraries and applications for better CPU features will usually get you no noticeable improvement. There is some noticeable improvement to be had by re-compiling the kernel to match your hardware.

User avatar
Puppus Dogfellow
Posts: 1667
Joined: Tue 08 Jan 2013, 01:39
Location: nyc

#22 Post by Puppus Dogfellow »

i've decided to try this even though i don't know what i'm doing....

gleaning/misreading that running

Code: Select all

src2pkg -CWD -PET --arch=i686 mixxx-1.11.0-src.tar.gz 
from a terminal in a folder containing mixxx-1.11.0-src.tar.gz would give me the pet i want, i get:

# src2pkg -CWD -PET --arch=i686 mixxx-1.11.0-src.tar.gz
Notice - Creating src2pkg-helpers:
src2pkg uses a shared library and a few programs
when creating packages. For best compatibility,
these binaries will be compiled on your system.
They are then installed in a private directory.
When done, src2pkg is ready for use.

Code: Select all

  TEMP_DIR=/usr/src/src2pkg/builds/src2pkg-helpers
  Starting build in 5 seconds
Unpacking sources - OK
Creating libsentry - OK
Creating tar-1.13 - OK
Creating coreutils - OK
Creating unionfs-fuse - OK
Creating pet package - Done 
Finished pet package is:
/usr/src/src2pkg/src2pkg-helpers/src2pkg-helpers-1.6-i486-1.pet
Installing pet package - 
--it's just hanging, so i figured i'd take this opportunity to ask what i've done wrong...(about a half hour so far on a moderately fast machine running precise 5.5.. lxtask shows no change in memory use and 0% cpu use).

thanks in advance.

(just checked the location the above helper pet is supposed to be in--scripts are there, but no pet.)

will try again...got a little further (helper pet created and installed) on a slower machine running 571, "build failed, must be a poor..."

:oops: forgot to copy the actual error message. (it also said something about there was no file with dependency information or something to that effect...sorry.)

attempt 2 with the 5.5. yielded no output. i hit enter with # src2pkg -CWD -PET --arch=i686 mixxx-1.11.0-src.tar.gz
and just got a cursor. back to the 571...

which gives me:

Code: Select all

# src2pkg -CWD -PET --arch=i686 mixx-1.11.0-src.tar.gz
FAILED! File: mixx-1.11.0-src.tar.gz not found!
Check for errors in the NAME, VERSION or SRC_SUFFIX. 
# src2pkg -CWD -PET --arch=i686 mixxx-1.11.0-src.tar.gz 
Found source archive: mixxx-1.11.0-src.tar.gz
Creating working directories:
   PKG_DIR=/initrd/mnt/dev_save/dwnldsfff/mix111/mixxx-1.11.0-i686-1
   SRC_DIR=/initrd/mnt/dev_save/dwnldsfff/mix111/mixxx-1.11.0-src-1
Unpacking source archive - Done
Correcting source permissions - Done
Checking for patches - None found
Found SConstruct file - Configuring using: 'scons configure prefix=/usr' Done
Compiling sources - Using: 'scons'
ERROR! Compiling source code has failed.
This usually happens because of missing libraries, or
badly written Makefiles or configure scripts.
Sorry! No Dependency or Requirements information found.

# 

live
Posts: 223
Joined: Wed 10 Feb 2010, 21:04

#23 Post by live »

i've decided to try this even though i don't know what i'm doing....
As a math teacher told me, people doing stats without understanding, is like playing with grenades :)

Now seriously,

Mixxx 1.10 is in PPM and needs about 94Mb of Qt4 dependicies.

--it's just hanging, so i figured i'd take this opportunity to ask what i've done wrong...
I experience the same, haven't try with mixxx but:
1/ run src2pkg cmd
2/ on prompting for installing the help pet: cancel.
3/ install the help pet.
4/ re-run src2pk cmd

src2pkg can not readily build everything, for example:
gedit-2.91.11.tar.gz
gcc 8.2
testdisk 7.0
SMPlayer 14
SeaMonkey

But can build others :)

User avatar
Puppus Dogfellow
Posts: 1667
Joined: Tue 08 Jan 2013, 01:39
Location: nyc

#24 Post by Puppus Dogfellow »

live wrote:
i've decided to try this even though i don't know what i'm doing....
As a math teacher told me, people doing stats without understanding, is like playing with grenades :)

Now seriously,

Mixxx 1.10 is in PPM and needs about 94Mb of Qt4 dependicies.


--it's just hanging, so i figured i'd take this opportunity to ask what i've done wrong...
I experience the same, haven't try with mixxx but:
1/ run src2pkg cmd
2/ on prompting for installing the help pet: cancel.
3/ install the help pet.
4/ re-run src2pk cmd

src2pkg can not readily build everything, for example:
gedit-2.91.11.tar.gz
gcc 8.2
testdisk 7.0
SMPlayer 14
SeaMonkey

But can build others :)
yeah, but on one machine i broke it and can't seem to repair it so i figured i'd try it this way...

will mess around some more later. thanks for the insight.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#25 Post by amigo »

The build for mixx is most likely failing because of missing the 'scons' program.
src2pkg can't do magic -there is too much variation in the methods used to configure and build software- as the scons example shows. src2pkg can sometimes create a correct package without any special options being passed at all. But there will always be packages which need special config options or scripted steps to build them.

I've used src2pkg to build more than 10,000 distinct sources and it allows enough flexibility to handle any sort of build -but a recipe or package-specific options are usually needed about 60% of the time. So don't be surprised or discouraged if it fails to do it automatically the first time. Many times, builds fail because of missing prerequisites.

User avatar
Puppus Dogfellow
Posts: 1667
Joined: Tue 08 Jan 2013, 01:39
Location: nyc

#26 Post by Puppus Dogfellow »

amigo wrote:The build for mixx is most likely failing because of missing the 'scons' program.
src2pkg can't do magic -there is too much variation in the methods used to configure and build software- as the scons example shows. src2pkg can sometimes create a correct package without any special options being passed at all. But there will always be packages which need special config options or scripted steps to build them.

I've used src2pkg to build more than 10,000 distinct sources and it allows enough flexibility to handle any sort of build -but a recipe or package-specific options are usually needed about 60% of the time. So don't be surprised or discouraged if it fails to do it automatically the first time. Many times, builds fail because of missing prerequisites.
it seems fairly amazing. i'm not discouraged at all.

:)

live
Posts: 223
Joined: Wed 10 Feb 2010, 21:04

#27 Post by live »

>amigo
All I meant, is what you said "src2pkg can't do magic" sometimes it works as is, some times not and gave some examples.

Sorry, if they were some misinterpretation.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#28 Post by amigo »

You can find lots of build scripts with patches, etc, for a reference implementation here:
http://distro.ibiblio.org/amigolinux/distro/kiss-4.0/

There are also many, many older scripts here:
http://distro.ibiblio.org/amigolinux/download/

You can have src2pkg generate a build script for you with:
src2pkg -N -Q tarball-name
or:
src2pkg -A tarball-name

The build scripts are easily modifiable to do anything at all. And they provide a repeatable, upgradeable recipe for later use. Provided that all needed depends are already present, most sources can be successfully compiled, to your needs, by only adding whatever options the sources configure script needs -although src2pkg handles the routine options like --prefix, --libdir, etc, automatically.

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

#29 Post by scsijon »

Very nice package amigo, that can save a lot of time for me with T290 building. There's about 5100 pets to create in all of the T2 V9.0 set and I was wondering how I was going to manage it all. I know I could just do those I need, but I would like to have all of them available.

Is there a simple way to :

-just point it at a directory and tell it to create pet's from all sources in that directory;

- output resultant pets to another named directory;

-error files to a third so I can just work on those with problems;

- creates a copy of the onscreen data into a single named file so that you can initially just leave it alone to run it's course, and later check back when working on a problem.

Oh yes, and could it maybe be told to deal with patch scripts if their in the same directory as the source package.

I know a tall order, but anything to make life easier.... :wink:

regards
scsijon

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#30 Post by amigo »

src2pkg lends itself well to being driven by other scripts -see the example GroupBuild scripts in the documentation. For best results, each source should be in its own sub-directory so that files generated or used by each build are kept separately. It also makes it possible to run more than one build at a time.

Any patches placed alongside the sources will be automatically applied.

Location of final packages can be controlled by setting PKG_DEST_DIR=/path/to/destination

The '-LOG' option will log the output of all important commands run during the build.

src2pkg is meant to create great packages -but *one at a time*. While it would be possible to make it run through a list of sources as a single run, it would defeat more important purposes. Irt can't do magic, so building 5,000+ sources is gonna take some work no matter what system you use.

While src2pkg can build lots of sources without any help or special options, there will always be some which need patches, special options or that must be done in a certain order. This is the same for all build systems, so at some point human-knowledge must come into play -which means writing build recipes, etc. However, src2pkg can leverage previous knowledge, by using rpm *.spec files, SlackBuild scrips or debian build files. This makes the process of creating a repo of sources for src2pkg much easier.

Take a good look at the example *.src2pkg scripts in the docs as there are examples for building inter-dependent packages and other unusual builds.

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

#31 Post by scsijon »

Yes, I agree, I've been having a play and also have found the src2pkg website, which is extremely helpfull.

I think my first problem will be an 'order-of-build' problem as I will need to have dependancies built and installed first before the packages needing them.

So I somehow need to list them, build them and install them before building the parent packages.

The -T I think will tell me if it needs a dependancy, thereby if so put it into build-later-list2, if not into build-first-list1.

I wonder what format it comes back in though, just the packagename or a lot of unwanted info? I think if it's useable like that, that is all I would need added if it's not there already in the switches, to complete an autobuild with dependancies.

This would autoloop with build-first-list1 being installed and build-later-list2 becoming the new source-list fed into the -T until there is nothing in either build* lists.

The -I will of course install the dependancy once built.

I shall play further, but I wonder if in the future it would be possible to build linux totally from source without the need to have interims such as T2, etc.

Seems like i'm back to scripting again, oh well :roll:

I think for now, i'll just build with src2pkg the minimum I need for T290 and work it from there. ie do the other packages from within a T290 base, not give up.

This leaves just which Kernel and it's settings left to choose!

Also, as a fyi, I just found that src2pkg --more-help, listed in the help, just errors out while src2pkg -hh is ok.

regards
scsijon

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

#32 Post by scsijon »

amigo, I'm having fun playing with this and do think it will be of much use to me, especially when I start to create all the extra T2 packages.

However, for now, I can't seem to find what output switch sets the final package as a $package.tar.bz2, can you help me please. It's needed for the way BarryK has setup the new T2 packaging system and i'd like to keep that part of it as the overall build seems to have become 'somewhat quicker'. It looks like it requires less post-build scripting work to happen than a pure .pet package does.

thanks
scsijon

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#33 Post by amigo »

You mean you want the final packages to have '.tar.bz2' suffix? And, that they use only a name without any ARCH or VERSION info in the name?
src2pkg does offer bzipped packages as an option, using '-TBZ'. However, you may find it quirky since nobody ses such packages. It was intended for use with vector linux IIRC. But, when you ask for a *.pet it will automatically set and use gzip compression with .pet suffix.

The best way to solve the above (either question), would be to write a small extension to src2pkg which would re-format and re-name the end result. You can create extension which are run at any and all steps of the build -either before or after the regular code of the step is run. You can see an explanation of the mechanism starting at line 2492 of the ChangeLog file in the docs and the feature must be enabled in the conf file -see notes at line 289. The extensions feature allows you to radically alter what src2pkg does, so it provides fertile ground for developing new features. Let me know if you still don't understand how to create and use them and I'll help you.

I like the T2 build system -but I don't like their package format as it is way too basic. I have developed a more usable package standard (actually two of them) which can be created by src2pkg -maybe you'd like to use it to build a base on?? The *.tpkg and *.tpm packages provide very rich package information -including package dependency info. The tpkg and tpm package management tools do not (yet) actually resolve depends for you (I am an old slacker, you know), however, the mechansim for doing so is quite simple *once the info is generated or specified and included in the package*. The nice thing about T2 is that it generates even BuildRequires info for you. I've done some work towards integrating this into src2pkg (for *.tpm packages). The run-time depends-generation already works great.

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

#34 Post by scsijon »

amigo wrote:The best way to solve the above (either question), would be to write a small extension to src2pkg which would re-format and re-name the end result. You can create extension which are run at any and all steps of the build -either before or after the regular code of the step is run. You can see an explanation of the mechanism starting at line 2492 of the ChangeLog file in the docs and the feature must be enabled in the conf file -see notes at line 289. The extensions feature allows you to radically alter what src2pkg does, so it provides fertile ground for developing new features. Let me know if you still don't understand how to create and use them and I'll help you..
ok, i'll have a look and a go before asking for help
amigo wrote:I like the T2 build system -but I don't like their package format as it is way too basic.
/cut/
The nice thing about T2 is that it generates even BuildRequires info for you. I've done some work towards integrating this into src2pkg (for *.tpm packages). The run-time depends-generation already works great.
Mageia does this also with their system, although sometimes there is too much info. given.

I have an idea this is what barryk was working towards using in his version of woof (not woof-ce) and his racy, which is T2. But when he went back to quirky (his favorite plaything) it was overtaken with another idea. He had a few notes about generation and integration some time ago on his blog, but they were only 'roughs'.

I'd really like to one day, just tell a working system that I want to add a new package by name, have it d/l the source, find dependancies from inside the package, check what dependancies I have already and if versions are sufficent, give me an option to go ahead or not, if it's a yes d/l missing or dependancies that need to be updated, build and install dependancies, build and install the new package and let me test it and either accept it or reject it and if rejected remove the package and all the new dependancies or step those that were already in back to the earlier versions. Pie in the sky at present, but it should happen one day.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#35 Post by amigo »

Your last paragraph describes exactly what nearly everyone wants -and what major distros implement to a great degree. Mageia is an rpm distro as is fedora and suse. debian/ubuntu & Co. use dpkg with *.deb packages. Both systems require a recipe for every single package which includes everything from the source url to the last script actions used to install the package. woof is completely different as it just uses pre-built packages.

There is no magic way to resolve dependencies properly. It requires the generation of a lot of accurate information which must be collected and collated into a useful form *at the time the package is created*, and then passed on *within the package* to package *management* tools.

generate, collect, collate/format -that's the hard part. This has to be done by the package *creation* software at package build-time.

The package *format* is the central concept of any package system. It must be capable of containing and transmitting all the info needed to carry out proper installation.

Dependency-resolution and proper up/down-grading, done of course by the package installation/mgmt tools, also depend on the use of known, sane repositories of packages. The repo-side of the equation also involves creating and making available information about the packages. repo creation and management is done by still another class of software.

There is one example extension (for vector linux in fact), here:
http://distro.ibiblio.org/amigolinux/do ... xtensions/

Try just creating a harmless extension with echo statements, so you can see if it is working and how/when.

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

64bit?

#36 Post by scsijon »

I'm going to be greedy for mick and ask when there will likely be a 64bit version for his slacko64?

please, pretty please.

:lol:

scsijon

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

extension?

#37 Post by scsijon »

I was wondering if it as possible to build a package from within a jailed (your term i think) system. Something along the lines of your -JAIL setting for an install, but extended to look for any dependancies there (say /var/tree_root) and not in the normal ('/') tree. You would need to set the tree base somewhere. Then as far as src2pkg was concerned, if the wanted prerequisite package or file is not already there it would fail or a version needs updating past what you had already built and installed in the jailed area.
It would mean that you would need a basic filesystem structure (/bin, /lib, /etc, ...) of course. In our case an opened puppy...sfs should suffice, but there is no reason something like aboriginal linux shouldn't be able to provide this structure in it's package.
It would at least then give you the ability to chroot for testing or even (maybe) develop for one system on top of another.

I hope you understand what i mean from this.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#38 Post by amigo »

As for a 64-bit version, I would try installing src2pkg normally and running 'src2pkg --setup', but I'm not sure that will compile correctly on *puppy* 64-but -it does work on Slackware 64-bit. On puppy, it might possibly compile for 64-bit usage but be named as a 32-bit.

As to using jails, that should work if the jail is created and used properly. If you chroot into the 'jailed' directory struvture, then references to the database path would reference that path within the jail. If you need to reference paths outside the 'jail', then you'd need to 'mount --bind ...' the needed items inside the jail.

scsijon
Posts: 1596
Joined: Thu 24 May 2007, 03:59
Location: the australian mallee
Contact:

#39 Post by scsijon »

amigo wrote:As for a 64-bit version, I would try installing src2pkg normally and running 'src2pkg --setup', but I'm not sure that will compile correctly on *puppy* 64-but -it does work on Slackware 64-bit. On puppy, it might possibly compile for 64-bit usage but be named as a 32-bit.

As to using jails, that should work if the jail is created and used properly. If you chroot into the 'jailed' directory struvture, then references to the database path would reference that path within the jail. If you need to reference paths outside the 'jail', then you'd need to 'mount --bind ...' the needed items inside the jail.
I tried with barry's april64 (with devx) and it installed but failed miserably with the --setup so I didn't go further for the moment. Partly due I think to things like april uses a few packages that are later than the current src2pkg so maybe a new version is due.

regards

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#40 Post by amigo »

What was the output from 'src2pkg --setup'?

Post Reply