[ANNOUNCE] src2pkg-2.1 released

[amigo]

Thanks for using src2pkg. I appreciate the feedback and was hoping to answer at least some of your questions today, but it will have to wait till tomorrow.

[big_bass]

Hey Pizzasgood !

good to see you that you are doing a source build with tgz packages also

the format is well established any custom packages you make
will be very linux friendly

But when I'm doing the chrooted phase of my build, that isn't available. I could compile it ahead of time,

you know that if you use slackwares pkgtool you could eliminate many of those problems you encountered by building the packages then

do your installpkg -root (into your new chroot environment )

easy and error free with those tricky packages


thanks also for all the feed back tommorow Ill look over all your comments and check out the patches thanks for posting them


Joe

[amigo]

The first thing I noticed, testing in Puppy, was that the -C option doesn't work with -PET and -PET2. Some quick grepping shows that 15-make_package_pet does not use $PKG_DEST_DIR anywhere, so I figure that's the problem.

This is an oversight -I was adding support for 5 new package formats at once. But, it works here because I have all the pertinent optikns et in the conf file to use CWD for everything. Good catch!

I didn't bother looking into it beyond that because I don't intend to use the .pet support anyway (I'm going with .tgz for now, and might come up with my own format later if that isn't good enough - binary packages aren't a big part of this project, I mainly just need to compile things and then log what goes where when they are installed).

Very good to use a better package format. I'll post an attachment later with my hacked version of pkgtools which uses a more advanced database which supprts/is supported by the advanced database features of src2pkg -including compatibility with slapt-get/gslapt. I also have tools which generate a compile-order list from the package database.

"The second thing I noticed is that it doesn't have support for aufs. Normally in Puppy that's fine because it seems to use the fuse deal instead. But when I'm doing the chrooted phase of my build, that isn't available. I could compile it ahead of time, and probably will in the long run, but for now, I just rely on the native unionfs or aufs support in the host system's kernel. Since the version of Puppy I'm running uses aufs, src2pkg won't work with the -UNION option unless I patch it. It's a simple enough change that I'm wondering whether there is perhaps a reason it doesn't already support aufs? Is it known to be unstable when used with src2pkg? Or was this just an oversight? I've attached the patch I made, and have been using this to build my system and so far haven't really had any problems."

Patches wil get you everywhere...
src2pkg uses either unionfs-fuse(the unionfs executable in /usr/libexec/src2pkg/bin or the real unionfs kernel module. The unionfs exe uses fuse -you could compile fuse and have it include building the kernel module for you -the code is all in the sources.
The contributor who worked out the original code only used the unionfs-fuse program -I added code to be able to use the real unionfs. The syntax is different. The unionfs prog is built by default when you run 'src2pkg --setup' if you have fuse installed and it will be used if found first on your system -but you can also specify to use real unionfs in your cond file. Your patch provides syntax for using aufs, so I'll make that another option in the conf file.

I did have an issue after installing glibc, but I'm not sure if that's due to aufs or not. It compiled and installed fine, but then something broke and anything that tried to view my active mounts (like running the "mount" command, or running "df") would segfault, and trying to view regions of the filesystem involved with the build would cause weird things to happen. I had a kernel oops in /var/log/messages about include/linux/dcache.h:324. But after a reboot everything was happy again, and no other packages had this problem (glibc continued to have it any time I tried again). I haven't had a chance yet to try compiling glibc from a native non-aufs system yet to compare against.

Using a real hard-disk installation is always best -especially for low-level stuff like glibc. To upgrade glibc on a running system, you need to go into runlevel 1 'maintenance mode' so that nothing which is using glibc is running. Even then, you may find that things are weird until you reboot.

Third thing: I'm confused as to the purpose of $EXCL_DIR in 08D-union_root_install. Judging by the name, I would guess that it's meant to hide directories that you don't want to appear in the unioned filesystem. But that doesn't make sense to me, because the mount command has $EXCL_DIR at the bottom of the stack. When I tried building, I got errors relating to /dev, /proc, etcetera. I tried reordering it to put $EXCL_DIR between $PKG_DIR and / instead, and for good measure I added .wh.__dir_opaque files in each of the directories that were created in $EXCL_DIR, and then it worked. I attached the patch so you can better see what I did. Maybe I'm missing something? Or maybe unionfs via fuse handles the layering differently from how aufs does it.

Yes, it does. And unionfs is probably not exactly the same as aufs. If you've got aufs working there with the patch, it should be trivial to incorporate aufs functionality.

"Fourth thing: When given an unrecognized option, src2pkg exits with an error code of 0. Is that intentional? I attached a patch that tells it to use "exit 1" instead. This way if there's a glitch in one of my scripts so that they try passing something bogus to src2pkg an error will be detected.

I only recently worked on getting src2pkg to exit with intelligent status from wherever -glad you found a missing spot.

"Fifth thing: Lots of packages provide a "make check" rule, but there doesn't seem to be a good way to run this short of making a .auto script and inserting it, which duplicates a lot of code. So I made a patch that adds a 'verify_build' function (in the /usr/libexec/src2pkg/07A-verify_build file). This function will run "$DEFAULT_MAKE_COMMAND check" or whatever has been defined in $VERIFY_COMMAND, logging the results if logging is enabled, and displaying a warning if it doesn't return 0. There is also an $EXIT_ON_VERIFY_FAILURE variable that when set to "YES" will tell it to exit if there is a failure. Running the check can be enabled by passing the -VER option to src2pkg (or setting VERIFY_BUILD="YES"), and $VERIFY_COMMAND can be set with the -t= or --test_build= option (which implies -VER). I've provided the patch, and also a separate patch for the man file (when it's decompressed). There are probably some example files or something that also need patching, and maybe the sample configuration file. I haven't actually gotten around to updating my build system to take advantage of this yet, so it hasn't been heavily tested. It does seem to work though.

src2pkg tries to be as generic as possible in its' defaults -anything outside those boundaries is worthy of documentation -that was the first emphasis of the src2pkg API. You can always use:
MAKE_COMMAND='make all check'
Same for sequences like:
MAKE_COMMAND='make depend all'
That said, I'll definitely look hard at the patch and try to get the feature incorporated to work smoothly.

Sixth issue: Some packages (I forget the one I had this problem with) have their configuration in a subdirectory, but shouldn't actually be built from it. They want to be built from their main area with some options given to make (I think it was "make -c src"). The problem is that src2pkg sees the subdirectory and just dives in automatically, so that later when the makefile refers to something in the main area via a "..", it doesn't located it and errors (IIRC, at this point it was in the bind-mounted OBJ_DIR in the unioned environment). My quickfix for that was to edit my .auto script to "touch $SRC_DIR/Makefile" so that src2pkg would stay put. It would probably be better to have a commandline option or flag variable to tell it not to go into a subdirectory even if it finds one. So far I've only had this issue with the one package, so I haven't gotten around to implementing that.

I think I see an easy way to implement the 'no-dive' part -src2pkg has the '-s=??' option which lets you specify the subdir if it is not detected. Adding a 'none' option or inetgration with an existing option for cmake builds of USE_TOP_DIR.
Still, you can always get around one-off cases by commenting the 'offending' lines in the src2pkg build script and inserting code like:
#compile_source
cd $SRC_DIR
make -c src
BTW, it is a good practice to change the name of any build scripts which contain manual code or special options -dropping the .auto suffix. This keeps the scripts from being clobbered if you accidentally run src2pkg -N/-A in that directory.

"Seventh thing: When using -JAIL instead of -UNION, it seems that programs that would have otherwise installed over a symlink to busybox (for example, /usr/bin/realpath), the package it creates includes /bin/busybox (but is otherwise normal and does include the actual file that it should have). Maybe that's one of the known flaws with the -JAIL method. Figured I'd point it out just in case.

Hmm, that sounds odd. Anytime you get strange or unexpected results, try building the package using one of the other INSTALL_TYPE options. The most accurate and safe is SAFE. REAL is the same thing but no backups/restoration is done! 100% invasive. UNION and JAIL will both occassionally fail. DESTDIR is the least reliable, but still most packages can be built as a normal user using the DEST option. The UNION option is, so far, more reliable than JAIL.

If you can give me an example source which is doing this, it would help. This sounds like a missing '-a' option to the 'cp' command somewhere... I'll have a look.

Very timely that you have brought these things up as I am trying to reach a 'Summer Release' point -my garden and my 5-year-old beckon me...

email me at: amigo AT ibiblio DOT org
so I can have your name for the src2pkg ChangeLog. That way I can alos send you my pkgtools -there is still time/room to make some changes to the format which you might see as benficial.

[Pizzasgood]

Hey Pizzasgood !

good to see you that you are doing a source build with tgz packages also

the format is well established any custom packages you make
will be very linux friendly

Well, I'm putting very little energy into that. The only good reasons I can see (besides time) to support binary packages are for closed-source stuff and for embedded systems where it isn't practical to compile on the system itself. Well, and also for if you happen to have multiple systems that are all compiled the same way, so that you don't have to recompile multiple times.

Basically, I won't be offering many, if any, binary packages. The support will be there in case a user wants to create one for whatever reason, but I'm trying to operate under the assumption that everybody will have compiled their own unique system from scratch.

Also, my source-based format is pretty simple and small, so that auditing packages will be easy, and sharing packages is very low-bandwidth. With binary packages you have to trust the package maintainer a lot more. The situation in Puppy is a little scary. With a source based format there will be less room for the package maintainer to cause trouble without causing raised eyebrows. You would still need to trust the source from which the sourcefile is downloaded, of course. But that's easier to do. I would be more worried about a random person giving you an infected binary of, say, Vim, than about the Vim developers doing something fishy.

src2pkg tries to be as generic as possible in its' defaults -anything outside those boundaries is worthy of documentation -that was the first emphasis of the src2pkg API. You can always use:
MAKE_COMMAND='make all check'
Same for sequences like:
MAKE_COMMAND='make depend all'
That said, I'll definitely look hard at the patch and try to get the feature incorporated to work smoothly.

Thanks. I basically just copied the 07-compile_source file and hacked it into something that seemed like it worked the way I wanted. There's probably a lot of extra stuff that isn't really applicable, but not being very familiar with src2pkg's internals I didn't want to strip out too much.

One reason for doing it this way instead of via MAKE_COMMAND is have the option to do the check and log the results without necessarily aborting due to an error. Some packages have known errors, but maybe not in a critical feature you care about. But you would still want to do the check in case other parts have an error.

I'm trying to minimize my use of src2pkg scripts, and only resort to them when more power is needed. For most packages I would like to have a single file like this:

Code: Select all

SOURCE_NAME="realpath-0.1.tar.gz"
EXTRA_CONFIGS="--enable-dohicky"
MD5=5f56e159ffde60a4a5edd71c1f2db297
URLS=(
    http://some_repo.com/pub/realpath-0.1.tar.gz
    http://some_mirror.com/pub/realpath-0.1.tar.gz
)
WEB=http://www.realpath_website.com/

More options can be added for stuff like PRE_FIX, downloading patches, etc., but it should be all static, no actual execution. Those files are just sourced, so they technically can have execution, but I'm going to try to detect that ahead of time and refuse to use the file. In the long run I'll probably change the code to parse the file instead of sourcing it, just to be safe. The src2pkg-related variables are then exported before running src2pkg itself, so that it sees them.

When files like that aren't powerful enough, a secondary package format is used. The other format is a tarball that can optionally include additional files (like icons or default configuration files) along with a full blown src2pkg script. But these are more trouble to analyze manually, so their use is discouraged when not necessary.

I want to make it very easy for a user to look in a package and find out what it does so they can be sure there is no funny business.

(I already wrote much of the code to support this system yesterday.)

If you can give me an example source which is doing this, it would help. This sounds like a missing '-a' option to the 'cp' command somewhere... I'll have a look.

Sure thing. I attached the simple realpath program I wrote.

[Sit Heel Speak]

...Saves me from having to reinvent a couple wheels...

Trying to comprehend this thread, to orient myself here...you are using DejaGnu a.k.a. runtest, aren't you? The definitive example being, how the Makefile of gcc runs its 6-to-12-hour-long make -k check.

[Pizzasgood]

You can throw out everything I said about AUFS. I was really using unionfs the whole time. I was correct in saying that my 4.1.2 installation was using aufs, but it also includes support for unionfs. The catch is that it's compiled as a module, so grepping /proc/filesystems doesn't show support for it unless the unionfs module is loaded. That happens automatically if you try to mount with "-t unionfs". But if, as src2pkg does, you first look for the capability, then you won't get anywhere.

The automatically loading unionfs part is what tricked me. I had already experimented (during a previous session) with the "mount -t unionfs" command, before I realized that Puppy was using aufs for the layers it was already using. So I knew that mounting with -t unionfs worked. When I remembered that this version of Puppy used aufs, and saw that /proc/filesystems did not list unionfs but did list aufs, I just assumed that aufs provided a compatibility layer so that you could use -t unionfs with it. But it doesn't. (For some reason I also didn't think that unionfs and aufs could coexist on the same system, but they actually can.)


So, to fix the issue I was having with src2pkg not working within the chrooted environment, I just need to make sure I modprobe unionfs before using it.

I'll see if I can find a different way for src2pkg to know whether unionfs is availible than by /proc/filesystems. Looking for a unionfs module in /lib could work for normal systems, but not through the chrooted environment. But "mount" somehow knows, so I'll see if I can find out how that works.


Oh, and regarding the glibc issue: I still haven't had a chance to do any testing. But when reading about aufs today, I found out that tmpfs eventually runs out of inodes and wraps around, which can confuse aufs. Now, as I discovered about ten minutes later, I wasn't actually using aufs. But maybe unionfs is also confused by the reused inodes. I had glibc run the check suite as well, when it was being installed, so maybe this inode reuse is where the problem came from. That would make sense, because the first time it happened, I noticed that the /tmp directory in the chroot had been replaced with a file that contained temporary data from my Firefox session (which was going on outside the chroot). Anyways, one of these days I'll have to do some experimenting to figure out what was going on.



As for the EXCL_DIR thing, I think I understand what the original point was now. It seems to have existed not to hide things at all, but to provide the mountpoints for the bindmount in case for some absurd reason they do not already exist in the / filesystem. It appears to be named "EXCL_DIR" because the directories created in it won't show up in the PKG_DIR, whereas if the "mkdir -p" commands had been run in the UNION_DIR those directories would show up there.

In that case, for the intended feature the order should be irrelevant.

But using it with unionfs seems to not like it when it tries to union the /sys and /proc directories (unionfs-fuse is fine with that). It doesn't cause any problems, just prints an error message and continues. But reordering EXCL_DIR and placing whiteout files into it as I did in my patch prevents the errors. And since those directories will all be bindmounted later anyway, it doesn't hurt anything. The whiteout files don't do anything when unionfs-fuse is used, but they don't hurt anything either, and regardless they get covered up once the bindmounts are done.

So that patch seems to be okay.



@SHS: Yeah, DejaGnu is installed by the point where I run "make -k check". Src2pkg is a system that you basically run like this:
src2pkg somefile.tar.gz
And it will unpack and compile the source, and then create a .tgz package out of it for you. It has some options to let you add configure flags and stuff. But the vanilla release doesn't do a "make check". That's why I wrote a patch that would add that functionality.


Though, after having used it for a bit, I probably should have set EXIT_ON_VERIFY_FAILURE=YES by default, since checking is disabled by default anyway. Perhaps better would be to leave EXIT_ON_VERIFY_FAILURE="NO" and then change the default to run the check automatically - but only if it detects an appropriate rule in the Makefile. Both of those options would result in you being able to typically either stick with the default action, or only modify one of the options (don't test, or be strict and require a successful test). But as it is, in the cases where you don't use the defaults you usually have to modify two options - run the check and be strict.

[amigo]

PG, I still haven't gotten around to looking at your patches yet.
The reason that src2pkg looks at /proc/filesystems is because the feature unionfs(or aufs) feature my be *hard-linked* into the kernel insetad of as a module.
I think that getting aufs to work as an option will be easy enough -it won't hurt to have it for systems which have aufs but not unionfs. It may work better than the 'real' thing.
src2pkg already allows you to specify to use unionfs instead of the unionfs-fuse utility.

Still, the use of any layered filesystem is 'iffy' for heavy development, as is the use of tmpfs. 'make check' can be a very intensive job -so can compiling with tests, debug or profiling features(try seamonkey).

I think I'll be able to work in the ability to run 'make check' as an option. The default should probably be to *not* do it, allowing you to enable it from the command-line when wanted.

Have you discovered the ability to use extensions with src2pkg? You can create scraps of code which will be run just before and/or after each of the named src2pkg build steps. This is a good way to add features which are not part of the main program, or to develop new code which might be included. A good example might be including some of the BK code for generating *.desktop files when the package doesn't already include one.

[Pizzasgood]

The reason that src2pkg looks at /proc/filesystems is because the feature unionfs(or aufs) feature my be *hard-linked* into the kernel insetad of as a module.

I know. It just doesn't work if it was compiled as a module and has not been loaded, so I was trying to come up with a way to check for that possibility that would also work through a chroot. I haven't come up with anything, so I'll just make sure to modprobe unionfs before I do that stage. Not a big deal.

Doing something like this would be enough to cover both situations in a normal non-chrooted install I think:

Code: Select all

if [ "$(grep unionfs /proc/filesystems)" ] || [ "$(find "/lib/modules/$(uname --kernel-release)/" -type f -name "unionfs*")" ]; then
    echo "Found unionfs support";
fi

Still, the use of any layered filesystem is 'iffy' for heavy development, as is the use of tmpfs. 'make check' can be a very intensive job -so can compiling with tests, debug or profiling features(try seamonkey).

I'll keep that in mind.

I think I'll be able to work in the ability to run 'make check' as an option. The default should probably be to *not* do it, allowing you to enable it from the command-line when wanted.

Cool.

Have you discovered the ability to use extensions with src2pkg? You can create scraps of code which will be run just before and/or after each of the named src2pkg build steps. This is a good way to add features which are not part of the main program, or to develop new code which might be included. A good example might be including some of the BK code for generating *.desktop files when the package doesn't already include one.

That's based on home directories, right? I'll have to take a look. I was thinking of using that in case I need to do any goofy stuff that wouldn't be appropriate for src2pkg in general. It might be nice to have a global area for extensions, but maybe with an option for the user to disable them in case they want vanilla src2pkg behavior. Or maybe do it the other way, so that src2pkg will not surprise them if there are extensions they aren't aware of.

[amigo]

I have integrated the 'make check' feature for you.

There is code in 08-fake_install which modprobes the fuse module for use with unionfs-fuse -I'll put a unionfs module routine there as well.

Yeah, extensions are looked for in the users $HOME/.src2pkg/extensions directory which must be created manually. This could be handled differently -the subsytem was built with a mind to security, so that admins can allow/deny users.
There's an example extenssion here:
http://distro.ibiblio.org/pub/linux/dis ... ns/vector/
I'm not aware of anyone having used the extensions before, but they are exactly for what you describe. Still, any code/ideas which might be useful to others usually gets placed in the main code. When I release the new version, you'll see that I implemented the 'make check' in a more compact and integrated manner -the main thing is that it not break the API. This feature, like hundreds of others fits nicely within the existing list of 16 instructions -hooked into running just after the main compile commands. The syntax is the same as your proposal, except it uses -CHECK instead of -VER (VERIFY is an already-existing variable internally). I also left out the optional use of a custom-written function -it's doubtful that this would be used.

What about aufs? Pretty easy to work in a few lines of separate code to check proc/filesystems or load the module and use distinct syntax if it is different from aufs.

[Pizzasgood]

Cool, thanks.

I didn't try aufs with src2pkg, but when I was playing with it manually it didn't like mounting /. So that might take some fiddling.

[big_bass]

Hey Pizzasgood

here's an example of how
sharing work on different bases will work even if you dont "host tgz packages" the method is the same work can be shared if a build script is made and even better with patches



@Hey amigo this is for you also you can you generate a src2pkg build with it

Hey
@SHS
@aragon

*read the build script it gives details of what was edited to build the package
I added some notes to it


Joe