Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 15 Sep 2014, 21:09
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
[ANNOUNCE] src2pkg-2.1 released
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 2 of 3 [31 Posts]   Goto page: Previous 1, 2, 3 Next
Author Message
aragon

Joined: 15 Oct 2007
Posts: 1698
Location: Germany

PostPosted: Sat 13 Feb 2010, 11:19    Post subject:  

amigo wrote:
Say aragon, since you are just around the corner from me, perhaps you'd like to get together over coffee some weekend?

yes we should. next 2 weeks are busy, but then it should be better. i'll send you a pm.
Quote:
I'd give a pretty Penny to make such a date with Joe -though I'd rather be around the corner from him instead of the other way around... AYYYY! Jalisco!

i think i understand that one Laughing

aragon

_________________
PUPPY SEARCH: http://wellminded.com/puppy/pupsearch.html
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Sat 13 Feb 2010, 11:43    Post subject:  

Great, I'll be expecting your PM.

I lived in Mexico for 15 years and miss it a lot, so I'm quite jealous of Joe being there... We are friends even at this distance, but if I were still there I'll bet we'd 'paint the town red' -uh have a great time, I mean.

It'll be interesting to meet a Puppian, aragon -I only have one acquaintance here who even knows what Linux is.
Back to top
View user's profile Send private message 
big_bass

Joined: 13 Aug 2007
Posts: 1747

PostPosted: Sat 13 Feb 2010, 22:00    Post subject:  

Hey Gilbert

I have about 13 years In Mexico
so my "Gringo Spanish" has improved a lot

a lot of good people here but "nada" for linux users
around this "barrio"
maybe in the the next "barrio" I hope

hey working on package tools giving it an Xdialog face lift
when I get it all together its looking good I 'll post a copy

being careful not to disturb that dialog monster
too much


Joe

_________________
debian wheezy ,linux mint, slackware I use them all and they all have good points
Mint would be best for general users though
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Sun 14 Feb 2010, 05:58    Post subject:  

Yeah, pkgtools is a mess. It would be nice to make it work using either dialog or Xdialog, but that would really take some work. I actually started some years ago on creating dialog/Xdialog 'widgets' as functions which could be used either way. The trouble is that the dimensions have to be tweaked when using dialog boxes with Xdialog.

I've been meaning to PM or email you with a copy of the latest hacked pkgtools I have. I think you'd find them to be more useful than the Slackware versions. I'm still (slowly) working on them and a probably gonna incorporate the optional use of spkg into them. spkg is a very fast clone of pkgtools written in C. Package removal is up to 30 times faster. I have hacked in most of the extra functionality which I have added to pktools themselves -like handling postrm scripts and the extra databse files *-required, *-supplies and others.

I have added a meta-data file to the pacage format which provides good info about each package -PACKAGER, date and system on which package was built. I have also written a script which generates a compile-order list, based on the info found in the pkg-required files.

I also wanted to 'flesh out' the reasons for suggesting that you use my altered versions of pktools:
I changed the location of the database from /var/log to /var/lib/pkgtools because this a.) conforms to the LFSH/LSB standards. b.) It allows you to then mount /var/log on tmpfs which is handy to keep /var/log from growing to huge sizes over time. Stuff in /var/log is really temporary info, but the package database is really semi-permanent info. c.) Putting the database in its' own location also reduces the clutter in /var/log because the new databse has several new directories for the new meta-data and other files. I have opted for having the info in various files (instead of a single file for each package or a single file for 'all* packages) in order to make them more readable and it also makes it much easier and cleaner to 'query' the database using grep and other such tools.

I have also commented on the weaknesses of the slitaz package format, but without elaborating. tazpkg's use of cpio is not good since cpio creates new directories with perms of 700 instead of 755, so any package that creates new directories would need to include post-install code which corrects the perms of the dirs. Also, tazpkg has no facility for a BUILD number or ARCH for your packages so it is pretty limited.

I'll try to put together a package for you of my hacked tools, if you like, so any fixes you make can be incorporated. BTW, you can easily convert to using the new database location by simply copying or moving the existing slack-style database files into the new location. Then, when you use src2pkg with the '-E' option, you'll get (and keep) those extra database files which provide the data needed to do dependency-resolution using slapt-get (or with some extra functionality to be added to pkgtools).
Ojala, y que te vaya bonito..
Back to top
View user's profile Send private message 
Pizzasgood


Joined: 04 May 2005
Posts: 6270
Location: Knoxville, TN, USA

PostPosted: Tue 23 Mar 2010, 20:34    Post subject:  

EDIT: Disregard everything I say about AUFS in this post. See my second post after this one for clarification about my confusion. And don't use the aufs patch I attached....

So, I'm currently using src2pkg as part of my automated build system for CheesyRamHog Linux, and intend to continue using it as part of a simple source-based package management system (compiling and doing a -UNION install, and then processing the package it outputs to track the files, for now).


The first thing I noticed, testing in Puppy, was that the -C option doesn't work with -PET and -PET2. Some quick grepping shows that 15-make_package_pet does not use $PKG_DEST_DIR anywhere, so I figure that's the problem. I didn't bother looking into it beyond that because I don't intend to use the .pet support anyway (I'm going with .tgz for now, and might come up with my own format later if that isn't good enough - binary packages aren't a big part of this project, I mainly just need to compile things and then log what goes where when they are installed).


The second thing I noticed is that it doesn't have support for aufs. Normally in Puppy that's fine because it seems to use the fuse deal instead. But when I'm doing the chrooted phase of my build, that isn't available. I could compile it ahead of time, and probably will in the long run, but for now, I just rely on the native unionfs or aufs support in the host system's kernel. Since the version of Puppy I'm running uses aufs, src2pkg won't work with the -UNION option unless I patch it. It's a simple enough change that I'm wondering whether there is perhaps a reason it doesn't already support aufs? Is it known to be unstable when used with src2pkg? Or was this just an oversight? I've attached the patch I made, and have been using this to build my system and so far haven't really had any problems. I did have an issue after installing glibc, but I'm not sure if that's due to aufs or not. It compiled and installed fine, but then something broke and anything that tried to view my active mounts (like running the "mount" command, or running "df") would segfault, and trying to view regions of the filesystem involved with the build would cause weird things to happen. I had a kernel oops in /var/log/messages about include/linux/dcache.h:324. But after a reboot everything was happy again, and no other packages had this problem (glibc continued to have it any time I tried again). I haven't had a chance yet to try compiling glibc from a native non-aufs system yet to compare against.


Third thing: I'm confused as to the purpose of $EXCL_DIR in 08D-union_root_install. Judging by the name, I would guess that it's meant to hide directories that you don't want to appear in the unioned filesystem. But that doesn't make sense to me, because the mount command has $EXCL_DIR at the bottom of the stack. When I tried building, I got errors relating to /dev, /proc, etcetera. I tried reordering it to put $EXCL_DIR between $PKG_DIR and / instead, and for good measure I added .wh.__dir_opaque files in each of the directories that were created in $EXCL_DIR, and then it worked. I attached the patch so you can better see what I did. Maybe I'm missing something? Or maybe unionfs via fuse handles the layering differently from how aufs does it.


Fourth thing: When given an unrecognized option, src2pkg exits with an error code of 0. Is that intentional? I attached a patch that tells it to use "exit 1" instead. This way if there's a glitch in one of my scripts so that they try passing something bogus to src2pkg an error will be detected.


Fifth thing: Lots of packages provide a "make check" rule, but there doesn't seem to be a good way to run this short of making a .auto script and inserting it, which duplicates a lot of code. So I made a patch that adds a 'verify_build' function (in the /usr/libexec/src2pkg/07A-verify_build file). This function will run "$DEFAULT_MAKE_COMMAND check" or whatever has been defined in $VERIFY_COMMAND, logging the results if logging is enabled, and displaying a warning if it doesn't return 0. There is also an $EXIT_ON_VERIFY_FAILURE variable that when set to "YES" will tell it to exit if there is a failure. Running the check can be enabled by passing the -VER option to src2pkg (or setting VERIFY_BUILD="YES"), and $VERIFY_COMMAND can be set with the -t= or --test_build= option (which implies -VER). I've provided the patch, and also a separate patch for the man file (when it's decompressed). There are probably some example files or something that also need patching, and maybe the sample configuration file. I haven't actually gotten around to updating my build system to take advantage of this yet, so it hasn't been heavily tested. It does seem to work though.


Sixth issue: Some packages (I forget the one I had this problem with) have their configuration in a subdirectory, but shouldn't actually be built from it. They want to be built from their main area with some options given to make (I think it was "make -c src"). The problem is that src2pkg sees the subdirectory and just dives in automatically, so that later when the makefile refers to something in the main area via a "..", it doesn't located it and errors (IIRC, at this point it was in the bind-mounted OBJ_DIR in the unioned environment). My quickfix for that was to edit my .auto script to "touch $SRC_DIR/Makefile" so that src2pkg would stay put. It would probably be better to have a commandline option or flag variable to tell it not to go into a subdirectory even if it finds one. So far I've only had this issue with the one package, so I haven't gotten around to implementing that.


Seventh thing: When using -JAIL instead of -UNION, it seems that programs that would have otherwise installed over a symlink to busybox (for example, /usr/bin/realpath), the package it creates includes /bin/busybox (but is otherwise normal and does include the actual file that it should have). Maybe that's one of the known flaws with the -JAIL method. Figured I'd point it out just in case.



Otherwise I like this. Saves me from having to reinvent a couple wheels.


EDIT: I should mention that if there is a good reason to not use aufs, that would be okay for me. CRH itself will be using UnionFS. It's just my host system (Puppy 4.1.2) that is using aufs, and I intend to be migrating over into CRH in the very near future (hopefully by the end of the week, because it would be a lot more painful to try to do that after school fires back up next week (it's Spring Break ATM)).
src2pkg-2.1-noarch-7-support_aufs.patch.gz
Description  Don't use this patch, it is incorrect.
gz

 Download 
Filename  src2pkg-2.1-noarch-7-support_aufs.patch.gz 
Filesize  362 Bytes 
Downloaded  225 Time(s) 
src2pkg-2.1-noarch-7-fix_exclusion.patch.gz
Description  Rearrange the $EXCL_DIR variable and put whiteout files in its subdirs.
gz

 Download 
Filename  src2pkg-2.1-noarch-7-fix_exclusion.patch.gz 
Filesize  450 Bytes 
Downloaded  231 Time(s) 
src2pkg-2.1-noarch-7-unrec_opt_exitcode.patch.gz
Description  Use "exit 1" when there is an unrecognized option.
gz

 Download 
Filename  src2pkg-2.1-noarch-7-unrec_opt_exitcode.patch.gz 
Filesize  459 Bytes 
Downloaded  231 Time(s) 
src2pkg-2.1-noarch-7-add_make_check.patch.gz
Description  Add -VER and -t|--test_build options for using "make check"
gz

 Download 
Filename  src2pkg-2.1-noarch-7-add_make_check.patch.gz 
Filesize  3.13 KB 
Downloaded  225 Time(s) 
src2pkg-2.1-noarch-7-add_make_check_man_page.patch.gz
Description  Update the (decompressed!) manpage to document the -VER and -t options I added.
gz

 Download 
Filename  src2pkg-2.1-noarch-7-add_make_check_man_page.patch.gz 
Filesize  472 Bytes 
Downloaded  229 Time(s) 

_________________
Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib


Last edited by Pizzasgood on Sat 27 Mar 2010, 15:09; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Wed 24 Mar 2010, 17:00    Post subject:  

Thanks for using src2pkg. I appreciate the feedback and was hoping to answer at least some of your questions today, but it will have to wait till tomorrow.
Back to top
View user's profile Send private message 
big_bass

Joined: 13 Aug 2007
Posts: 1747

PostPosted: Wed 24 Mar 2010, 23:41    Post subject:  

Hey Pizzasgood !

good to see you that you are doing a source build with tgz packages also Cool

the format is well established any custom packages you make
will be very linux friendly


Quote:

But when I'm doing the chrooted phase of my build, that isn't available. I could compile it ahead of time,


you know that if you use slackwares pkgtool you could eliminate many of those problems you encountered by building the packages then

do your installpkg -root (into your new chroot environment )

easy and error free with those tricky packages


thanks also for all the feed back tommorow Ill look over all your comments and check out the patches thanks for posting them


Joe

_________________
debian wheezy ,linux mint, slackware I use them all and they all have good points
Mint would be best for general users though
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Thu 25 Mar 2010, 10:48    Post subject:  

Quote:
The first thing I noticed, testing in Puppy, was that the -C option doesn't work with -PET and -PET2. Some quick grepping shows that 15-make_package_pet does not use $PKG_DEST_DIR anywhere, so I figure that's the problem.

This is an oversight -I was adding support for 5 new package formats at once. But, it works here because I have all the pertinent optikns et in the conf file to use CWD for everything. Good catch!
Quote:

I didn't bother looking into it beyond that because I don't intend to use the .pet support anyway (I'm going with .tgz for now, and might come up with my own format later if that isn't good enough - binary packages aren't a big part of this project, I mainly just need to compile things and then log what goes where when they are installed).

Very good to use a better package format. I'll post an attachment later with my hacked version of pkgtools which uses a more advanced database which supprts/is supported by the advanced database features of src2pkg -including compatibility with slapt-get/gslapt. I also have tools which generate a compile-order list from the package database.
Quote:

"The second thing I noticed is that it doesn't have support for aufs. Normally in Puppy that's fine because it seems to use the fuse deal instead. But when I'm doing the chrooted phase of my build, that isn't available. I could compile it ahead of time, and probably will in the long run, but for now, I just rely on the native unionfs or aufs support in the host system's kernel. Since the version of Puppy I'm running uses aufs, src2pkg won't work with the -UNION option unless I patch it. It's a simple enough change that I'm wondering whether there is perhaps a reason it doesn't already support aufs? Is it known to be unstable when used with src2pkg? Or was this just an oversight? I've attached the patch I made, and have been using this to build my system and so far haven't really had any problems."

Patches wil get you everywhere...
src2pkg uses either unionfs-fuse(the unionfs executable in /usr/libexec/src2pkg/bin or the real unionfs kernel module. The unionfs exe uses fuse -you could compile fuse and have it include building the kernel module for you -the code is all in the sources.
The contributor who worked out the original code only used the unionfs-fuse program -I added code to be able to use the real unionfs. The syntax is different. The unionfs prog is built by default when you run 'src2pkg --setup' if you have fuse installed and it will be used if found first on your system -but you can also specify to use real unionfs in your cond file. Your patch provides syntax for using aufs, so I'll make that another option in the conf file.

Quote:
I did have an issue after installing glibc, but I'm not sure if that's due to aufs or not. It compiled and installed fine, but then something broke and anything that tried to view my active mounts (like running the "mount" command, or running "df") would segfault, and trying to view regions of the filesystem involved with the build would cause weird things to happen. I had a kernel oops in /var/log/messages about include/linux/dcache.h:324. But after a reboot everything was happy again, and no other packages had this problem (glibc continued to have it any time I tried again). I haven't had a chance yet to try compiling glibc from a native non-aufs system yet to compare against.


Using a real hard-disk installation is always best -especially for low-level stuff like glibc. To upgrade glibc on a running system, you need to go into runlevel 1 'maintenance mode' so that nothing which is using glibc is running. Even then, you may find that things are weird until you reboot.

Quote:
Third thing: I'm confused as to the purpose of $EXCL_DIR in 08D-union_root_install. Judging by the name, I would guess that it's meant to hide directories that you don't want to appear in the unioned filesystem. But that doesn't make sense to me, because the mount command has $EXCL_DIR at the bottom of the stack. When I tried building, I got errors relating to /dev, /proc, etcetera. I tried reordering it to put $EXCL_DIR between $PKG_DIR and / instead, and for good measure I added .wh.__dir_opaque files in each of the directories that were created in $EXCL_DIR, and then it worked. I attached the patch so you can better see what I did. Maybe I'm missing something? Or maybe unionfs via fuse handles the layering differently from how aufs does it.

Yes, it does. And unionfs is probably not exactly the same as aufs. If you've got aufs working there with the patch, it should be trivial to incorporate aufs functionality.
Quote:

"Fourth thing: When given an unrecognized option, src2pkg exits with an error code of 0. Is that intentional? I attached a patch that tells it to use "exit 1" instead. This way if there's a glitch in one of my scripts so that they try passing something bogus to src2pkg an error will be detected.

I only recently worked on getting src2pkg to exit with intelligent status from wherever -glad you found a missing spot.
Quote:

"Fifth thing: Lots of packages provide a "make check" rule, but there doesn't seem to be a good way to run this short of making a .auto script and inserting it, which duplicates a lot of code. So I made a patch that adds a 'verify_build' function (in the /usr/libexec/src2pkg/07A-verify_build file). This function will run "$DEFAULT_MAKE_COMMAND check" or whatever has been defined in $VERIFY_COMMAND, logging the results if logging is enabled, and displaying a warning if it doesn't return 0. There is also an $EXIT_ON_VERIFY_FAILURE variable that when set to "YES" will tell it to exit if there is a failure. Running the check can be enabled by passing the -VER option to src2pkg (or setting VERIFY_BUILD="YES"), and $VERIFY_COMMAND can be set with the -t= or --test_build= option (which implies -VER). I've provided the patch, and also a separate patch for the man file (when it's decompressed). There are probably some example files or something that also need patching, and maybe the sample configuration file. I haven't actually gotten around to updating my build system to take advantage of this yet, so it hasn't been heavily tested. It does seem to work though.

src2pkg tries to be as generic as possible in its' defaults -anything outside those boundaries is worthy of documentation -that was the first emphasis of the src2pkg API. You can always use:
MAKE_COMMAND='make all check'
Same for sequences like:
MAKE_COMMAND='make depend all'
That said, I'll definitely look hard at the patch and try to get the feature incorporated to work smoothly.
Quote:

Sixth issue: Some packages (I forget the one I had this problem with) have their configuration in a subdirectory, but shouldn't actually be built from it. They want to be built from their main area with some options given to make (I think it was "make -c src"). The problem is that src2pkg sees the subdirectory and just dives in automatically, so that later when the makefile refers to something in the main area via a "..", it doesn't located it and errors (IIRC, at this point it was in the bind-mounted OBJ_DIR in the unioned environment). My quickfix for that was to edit my .auto script to "touch $SRC_DIR/Makefile" so that src2pkg would stay put. It would probably be better to have a commandline option or flag variable to tell it not to go into a subdirectory even if it finds one. So far I've only had this issue with the one package, so I haven't gotten around to implementing that.

I think I see an easy way to implement the 'no-dive' part -src2pkg has the '-s=??' option which lets you specify the subdir if it is not detected. Adding a 'none' option or inetgration with an existing option for cmake builds of USE_TOP_DIR.
Still, you can always get around one-off cases by commenting the 'offending' lines in the src2pkg build script and inserting code like:
#compile_source
cd $SRC_DIR
make -c src
BTW, it is a good practice to change the name of any build scripts which contain manual code or special options -dropping the .auto suffix. This keeps the scripts from being clobbered if you accidentally run src2pkg -N/-A in that directory.
Quote:

"Seventh thing: When using -JAIL instead of -UNION, it seems that programs that would have otherwise installed over a symlink to busybox (for example, /usr/bin/realpath), the package it creates includes /bin/busybox (but is otherwise normal and does include the actual file that it should have). Maybe that's one of the known flaws with the -JAIL method. Figured I'd point it out just in case.

Hmm, that sounds odd. Anytime you get strange or unexpected results, try building the package using one of the other INSTALL_TYPE options. The most accurate and safe is SAFE. REAL is the same thing but no backups/restoration is done! 100% invasive. UNION and JAIL will both occassionally fail. DESTDIR is the least reliable, but still most packages can be built as a normal user using the DEST option. The UNION option is, so far, more reliable than JAIL.

If you can give me an example source which is doing this, it would help. This sounds like a missing '-a' option to the 'cp' command somewhere... I'll have a look.

Very timely that you have brought these things up as I am trying to reach a 'Summer Release' point -my garden and my 5-year-old beckon me...

email me at: amigo AT ibiblio DOT org
so I can have your name for the src2pkg ChangeLog. That way I can alos send you my pkgtools -there is still time/room to make some changes to the format which you might see as benficial.
Quote:

Last edited by amigo on Thu 25 Mar 2010, 13:52; edited 1 time in total
Back to top
View user's profile Send private message 
Pizzasgood


Joined: 04 May 2005
Posts: 6270
Location: Knoxville, TN, USA

PostPosted: Thu 25 Mar 2010, 13:17    Post subject:  

Quote:
Hey Pizzasgood !

good to see you that you are doing a source build with tgz packages also Cool

the format is well established any custom packages you make
will be very linux friendly

Well, I'm putting very little energy into that. The only good reasons I can see (besides time) to support binary packages are for closed-source stuff and for embedded systems where it isn't practical to compile on the system itself. Well, and also for if you happen to have multiple systems that are all compiled the same way, so that you don't have to recompile multiple times.

Basically, I won't be offering many, if any, binary packages. The support will be there in case a user wants to create one for whatever reason, but I'm trying to operate under the assumption that everybody will have compiled their own unique system from scratch.

Also, my source-based format is pretty simple and small, so that auditing packages will be easy, and sharing packages is very low-bandwidth. With binary packages you have to trust the package maintainer a lot more. The situation in Puppy is a little scary. With a source based format there will be less room for the package maintainer to cause trouble without causing raised eyebrows. You would still need to trust the source from which the sourcefile is downloaded, of course. But that's easier to do. I would be more worried about a random person giving you an infected binary of, say, Vim, than about the Vim developers doing something fishy.


Quote:
src2pkg tries to be as generic as possible in its' defaults -anything outside those boundaries is worthy of documentation -that was the first emphasis of the src2pkg API. You can always use:
MAKE_COMMAND='make all check'
Same for sequences like:
MAKE_COMMAND='make depend all'
That said, I'll definitely look hard at the patch and try to get the feature incorporated to work smoothly.

Thanks. I basically just copied the 07-compile_source file and hacked it into something that seemed like it worked the way I wanted. There's probably a lot of extra stuff that isn't really applicable, but not being very familiar with src2pkg's internals I didn't want to strip out too much.

One reason for doing it this way instead of via MAKE_COMMAND is have the option to do the check and log the results without necessarily aborting due to an error. Some packages have known errors, but maybe not in a critical feature you care about. But you would still want to do the check in case other parts have an error.

I'm trying to minimize my use of src2pkg scripts, and only resort to them when more power is needed. For most packages I would like to have a single file like this:
Code:
SOURCE_NAME="realpath-0.1.tar.gz"
EXTRA_CONFIGS="--enable-dohicky"
MD5=5f56e159ffde60a4a5edd71c1f2db297
URLS=(
    http://some_repo.com/pub/realpath-0.1.tar.gz
    http://some_mirror.com/pub/realpath-0.1.tar.gz
)
WEB=http://www.realpath_website.com/

More options can be added for stuff like PRE_FIX, downloading patches, etc., but it should be all static, no actual execution. Those files are just sourced, so they technically can have execution, but I'm going to try to detect that ahead of time and refuse to use the file. In the long run I'll probably change the code to parse the file instead of sourcing it, just to be safe. The src2pkg-related variables are then exported before running src2pkg itself, so that it sees them.

When files like that aren't powerful enough, a secondary package format is used. The other format is a tarball that can optionally include additional files (like icons or default configuration files) along with a full blown src2pkg script. But these are more trouble to analyze manually, so their use is discouraged when not necessary.

I want to make it very easy for a user to look in a package and find out what it does so they can be sure there is no funny business.

(I already wrote much of the code to support this system yesterday.)


Quote:
If you can give me an example source which is doing this, it would help. This sounds like a missing '-a' option to the 'cp' command somewhere... I'll have a look.

Sure thing. I attached the simple realpath program I wrote.
realpath-0.1.tar.gz
Description  Installing this with src2pkg -JAIL will include /bin/busybox in the package.
gz

 Download 
Filename  realpath-0.1.tar.gz 
Filesize  1018 Bytes 
Downloaded  229 Time(s) 

_________________
Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib

Back to top
View user's profile Send private message Visit poster's website 
Sit Heel Speak


Joined: 30 Mar 2006
Posts: 2595
Location: downwind

PostPosted: Fri 26 Mar 2010, 12:13    Post subject:  

Pizzasgood wrote:
...Saves me from having to reinvent a couple wheels...
Trying to comprehend this thread, to orient myself here...you are using DejaGnu a.k.a. runtest, aren't you? The definitive example being, how the Makefile of gcc runs its 6-to-12-hour-long make -k check.
Back to top
View user's profile Send private message 
Pizzasgood


Joined: 04 May 2005
Posts: 6270
Location: Knoxville, TN, USA

PostPosted: Sat 27 Mar 2010, 14:59    Post subject:  

You can throw out everything I said about AUFS. I was really using unionfs the whole time. I was correct in saying that my 4.1.2 installation was using aufs, but it also includes support for unionfs. The catch is that it's compiled as a module, so grepping /proc/filesystems doesn't show support for it unless the unionfs module is loaded. That happens automatically if you try to mount with "-t unionfs". But if, as src2pkg does, you first look for the capability, then you won't get anywhere.

The automatically loading unionfs part is what tricked me. I had already experimented (during a previous session) with the "mount -t unionfs" command, before I realized that Puppy was using aufs for the layers it was already using. So I knew that mounting with -t unionfs worked. When I remembered that this version of Puppy used aufs, and saw that /proc/filesystems did not list unionfs but did list aufs, I just assumed that aufs provided a compatibility layer so that you could use -t unionfs with it. But it doesn't. (For some reason I also didn't think that unionfs and aufs could coexist on the same system, but they actually can.)


So, to fix the issue I was having with src2pkg not working within the chrooted environment, I just need to make sure I modprobe unionfs before using it.

I'll see if I can find a different way for src2pkg to know whether unionfs is availible than by /proc/filesystems. Looking for a unionfs module in /lib could work for normal systems, but not through the chrooted environment. But "mount" somehow knows, so I'll see if I can find out how that works.


Oh, and regarding the glibc issue: I still haven't had a chance to do any testing. But when reading about aufs today, I found out that tmpfs eventually runs out of inodes and wraps around, which can confuse aufs. Now, as I discovered about ten minutes later, I wasn't actually using aufs. But maybe unionfs is also confused by the reused inodes. I had glibc run the check suite as well, when it was being installed, so maybe this inode reuse is where the problem came from. That would make sense, because the first time it happened, I noticed that the /tmp directory in the chroot had been replaced with a file that contained temporary data from my Firefox session (which was going on outside the chroot). Anyways, one of these days I'll have to do some experimenting to figure out what was going on.



As for the EXCL_DIR thing, I think I understand what the original point was now. It seems to have existed not to hide things at all, but to provide the mountpoints for the bindmount in case for some absurd reason they do not already exist in the / filesystem. It appears to be named "EXCL_DIR" because the directories created in it won't show up in the PKG_DIR, whereas if the "mkdir -p" commands had been run in the UNION_DIR those directories would show up there.

In that case, for the intended feature the order should be irrelevant.

But using it with unionfs seems to not like it when it tries to union the /sys and /proc directories (unionfs-fuse is fine with that). It doesn't cause any problems, just prints an error message and continues. But reordering EXCL_DIR and placing whiteout files into it as I did in my patch prevents the errors. And since those directories will all be bindmounted later anyway, it doesn't hurt anything. The whiteout files don't do anything when unionfs-fuse is used, but they don't hurt anything either, and regardless they get covered up once the bindmounts are done.

So that patch seems to be okay.



@SHS: Yeah, DejaGnu is installed by the point where I run "make -k check". Src2pkg is a system that you basically run like this:
src2pkg somefile.tar.gz
And it will unpack and compile the source, and then create a .tgz package out of it for you. It has some options to let you add configure flags and stuff. But the vanilla release doesn't do a "make check". That's why I wrote a patch that would add that functionality.


Though, after having used it for a bit, I probably should have set EXIT_ON_VERIFY_FAILURE=YES by default, since checking is disabled by default anyway. Perhaps better would be to leave EXIT_ON_VERIFY_FAILURE="NO" and then change the default to run the check automatically - but only if it detects an appropriate rule in the Makefile. Both of those options would result in you being able to typically either stick with the default action, or only modify one of the options (don't test, or be strict and require a successful test). But as it is, in the cases where you don't use the defaults you usually have to modify two options - run the check and be strict.

_________________
Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib

Back to top
View user's profile Send private message Visit poster's website 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Sun 28 Mar 2010, 11:01    Post subject:  

PG, I still haven't gotten around to looking at your patches yet.
The reason that src2pkg looks at /proc/filesystems is because the feature unionfs(or aufs) feature my be *hard-linked* into the kernel insetad of as a module.
I think that getting aufs to work as an option will be easy enough -it won't hurt to have it for systems which have aufs but not unionfs. It may work better than the 'real' thing.
src2pkg already allows you to specify to use unionfs instead of the unionfs-fuse utility.

Still, the use of any layered filesystem is 'iffy' for heavy development, as is the use of tmpfs. 'make check' can be a very intensive job -so can compiling with tests, debug or profiling features(try seamonkey).

I think I'll be able to work in the ability to run 'make check' as an option. The default should probably be to *not* do it, allowing you to enable it from the command-line when wanted.

Have you discovered the ability to use extensions with src2pkg? You can create scraps of code which will be run just before and/or after each of the named src2pkg build steps. This is a good way to add features which are not part of the main program, or to develop new code which might be included. A good example might be including some of the BK code for generating *.desktop files when the package doesn't already include one.
Back to top
View user's profile Send private message 
Pizzasgood


Joined: 04 May 2005
Posts: 6270
Location: Knoxville, TN, USA

PostPosted: Tue 30 Mar 2010, 21:53    Post subject:  

Quote:
The reason that src2pkg looks at /proc/filesystems is because the feature unionfs(or aufs) feature my be *hard-linked* into the kernel insetad of as a module.
I know. It just doesn't work if it was compiled as a module and has not been loaded, so I was trying to come up with a way to check for that possibility that would also work through a chroot. I haven't come up with anything, so I'll just make sure to modprobe unionfs before I do that stage. Not a big deal.

Doing something like this would be enough to cover both situations in a normal non-chrooted install I think:
Code:
if [ "$(grep unionfs /proc/filesystems)" ] || [ "$(find "/lib/modules/$(uname --kernel-release)/" -type f -name "unionfs*")" ]; then
    echo "Found unionfs support";
fi



Quote:
Still, the use of any layered filesystem is 'iffy' for heavy development, as is the use of tmpfs. 'make check' can be a very intensive job -so can compiling with tests, debug or profiling features(try seamonkey).

I'll keep that in mind.

Quote:
I think I'll be able to work in the ability to run 'make check' as an option. The default should probably be to *not* do it, allowing you to enable it from the command-line when wanted.

Cool.

Quote:
Have you discovered the ability to use extensions with src2pkg? You can create scraps of code which will be run just before and/or after each of the named src2pkg build steps. This is a good way to add features which are not part of the main program, or to develop new code which might be included. A good example might be including some of the BK code for generating *.desktop files when the package doesn't already include one.

That's based on home directories, right? I'll have to take a look. I was thinking of using that in case I need to do any goofy stuff that wouldn't be appropriate for src2pkg in general. It might be nice to have a global area for extensions, but maybe with an option for the user to disable them in case they want vanilla src2pkg behavior. Or maybe do it the other way, so that src2pkg will not surprise them if there are extensions they aren't aware of.

_________________
Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib

Back to top
View user's profile Send private message Visit poster's website 
amigo

Joined: 02 Apr 2007
Posts: 2247

PostPosted: Wed 31 Mar 2010, 06:55    Post subject:  

I have integrated the 'make check' feature for you.

There is code in 08-fake_install which modprobes the fuse module for use with unionfs-fuse -I'll put a unionfs module routine there as well.

Yeah, extensions are looked for in the users $HOME/.src2pkg/extensions directory which must be created manually. This could be handled differently -the subsytem was built with a mind to security, so that admins can allow/deny users.
There's an example extenssion here:
http://distro.ibiblio.org/pub/linux/distributions/amigolinux/download/src2pkg/src2pkg-extensions/vector/
I'm not aware of anyone having used the extensions before, but they are exactly for what you describe. Still, any code/ideas which might be useful to others usually gets placed in the main code. When I release the new version, you'll see that I implemented the 'make check' in a more compact and integrated manner -the main thing is that it not break the API. This feature, like hundreds of others fits nicely within the existing list of 16 instructions -hooked into running just after the main compile commands. The syntax is the same as your proposal, except it uses -CHECK instead of -VER (VERIFY is an already-existing variable internally). I also left out the optional use of a custom-written function -it's doubtful that this would be used.

What about aufs? Pretty easy to work in a few lines of separate code to check proc/filesystems or load the module and use distinct syntax if it is different from aufs.
Back to top
View user's profile Send private message 
Pizzasgood


Joined: 04 May 2005
Posts: 6270
Location: Knoxville, TN, USA

PostPosted: Wed 31 Mar 2010, 11:40    Post subject:  

Cool, thanks.

I didn't try aufs with src2pkg, but when I was playing with it manually it didn't like mounting /. So that might take some fiddling.

_________________
Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib

Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 2 of 3 [31 Posts]   Goto page: Previous 1, 2, 3 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1831s ][ Queries: 12 (0.0069s) ][ GZIP on ]