Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 02 Sep 2014, 16:32
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
Question about LD_PRELOAD, environment, masking an app
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [5 Posts]  
Author Message
matiasbatero


Joined: 11 Oct 2012
Posts: 60
Location: Mar del Plata, Argentina

PostPosted: Sat 30 Aug 2014, 17:43    Post subject:  Question about LD_PRELOAD, environment, masking an app
Subject description: Ideas to make a portable package system beetween multiple distributions
 

Hi!

I want to execute applications that have been built for another distros (like archlinux). I have all dependency tree packages, i merged all of them on a directory, including libc, all.

This full directory, have the next form:

Code:
full_package_with_deps
./etc
./usr
   ./bin  <-- here is my executable (elf)
   ./lib   <-- here is glib dep version, and ld-linker
   {...etc etc...}
./var


etc etc.

I need to fake the executable inside on "full_package_with_deps/usr/bin/app" to avoid the app to look on my current / filesystem, where my libs have different versions, and different paths.

To do that.. i'm using: Environment variables.

Code:
LD_PRELOAD="path_to_glib_version" + "ld-linux.. current package linker"
LD_LOAD_LIBRARY="path_to_usr_lib"
./usr/bin/app


---

The question is, there is a way to set a full custom environment, (not for sandboxing, isolation) in order to ensure the execution of any app, without using chroots, jails?

The main objetive of all of this, is to build a system that allows to run any app from any distro, using only packages from repos. Like a portable-bundle.

For example:
Fedora: X app --> ( 1 + 23 dep packages RPM)
Download all, extract, merge.. and set a custom enviroment, to mask executables to point at their own libs directory.

In my opinion, the last limitation is the version kernel for compatibility.
There is a inherent disadvantage: the weight of the app. (But, this can be avoided using a modern filesystem that uses copy-on-write.


----

Chroot evironments, are usefull, but it has some problems. For example, if you have an audio app running inside, and you want to connect throught jack, with another app running on host-system, or in another chroot jail. Is not possible, as far i know.

----

The restrictions to do this system is:
1) Use binaries (no compilation). Use all binaries that exists on repos.
2) Use system posix built-ins to run apps. (The objetive is portability)
3) Make integration transparent.
4) The host, and other target systems, cannot be modified in order to run portable apps.
5) No lost in perfomance execution.

---

I you have some ideas to help me to do this, you are welcome.
Also, i need your opinions, and some analysis: Good, and Bad things for you.

Regards.
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2134
Location: The Blue Marble

PostPosted: Sat 30 Aug 2014, 22:38    Post subject:  

Matias,

The forum has a lousy PM system. I replied to your PM and the reply got stuck in my "outbox" for days. Anyway, I'm publishing my response to the PM here - I hope you don't mind. If you do mind, let me know I'll remove this post.

---

Quote:
I want to build a system that allows, build portable packages using different distro-repositories.
This is going to be difficult. You just need to follow the discussion on skype 4.3 thread as an excellent example of the difficulty.

Quote:
1) In puppy SFS's are very common, but i don't know how it is implemented. For example, if you have a SFS with app, it is mixed with / directory, throught unionfs? ¿ ( layer1=/ + layer2=SFS ) ?
Yes. See http://distro.ibiblio.org/fatdog/web/faqs/filesystem.html.

Quote:
I read some comments, that is not possible, because is redundant to union / with itself. BUT, you said in some comment that the / is already a unionfs, you can append another layer over this directory.
Correct, you can add SFS to "/" if "/" is already a unionfs. Otherwise, you can't.
Quote:
If its correct, it is only true for puppy?
It is true of any OS that uses unionfs as "/". Puppy is only one of them, there are many others: Slax, AntiX, and a few others. Not sure about DebianDog.

Quote:
because i'm on Debian Squeeze now, and i can't append layers over /. (Maybe my / is not a unionfs , but i don't know how to set it)
You can't. Setting unionfs for "/" is done at boot time, in initrd. You can't do it when your system is already up and running.

Quote:
2) It's possible to load SFS without using / path? For example, using a custom folder to make unions?

Yes, very possible. If your kernel has unionfs/aufs module, then it is very simple. If it does not, but at least it has FUSE module, then you can use unionfs-fuse (slower, though).

Quote:
3) Chroot is another method to launch app's. But i think that it has problems. For example.. you chroot an audio app, and you have qjack in host.. jackd cannot see the process to request a connection. So integration could be problematic.
You can. But it must be done carefully - you must allow some files from the "host" to show up under chroot, too. This usually requires some sort of bind-mounts.

And you can combine SFS with chroot - load SFS under a certain mountpoint, and then do a unionfs to put r/w layer on top of that (e.g using tmpfs), and then chroot to that. Example: http://www.lightofdawn.org/wiki/wiki.cgi/RunSlackoInFatdog.

Quote:

Resume: My idea is:
> Build a package using any distro repo (with all dependency tree inside of course) avoiding to compile and make pkgs from source. (Only using available resources). (Glib can be a problem, but if it is included in package, can be resolved)
You have summarised succintly what "woof-next" from Woof-CE does Very Happy

Quote:

> Execute it transparently over any system.
> Avoid virtualization and chroot.
Not always possible if you want to avoid chroot. If you want to avoid chroot you can't bring the original distros glibc. This usually is not a problem, but if the packages requires newer glibc than the one installed on the host, it won't work.

Quote:
I want to know, what are the inherent limitations. If this can be applicable.
And what do you think about this, if you have an alternative idea, or help.
If you want to avoid compiling, then you'll have to go with chroot method. Make the SFS, create aufs layers on top of it, use bind-mounts to bring /tmp, /dev, /proc, /sys and a few other important directories (perhaps /var too) to it, then chroot. It isn't perfect, but it is close.

If you *do* want to compile, then the best way is to install everything to /opt/your-pkgname - including the libraries, and then set LD_LIBRARY_PATH to point to it. This is how libreoffice does it, this is how firefox does it - they have everything including the libraries in their own directory, all executed under a wrapper script. Thus, no chroot or aufs needed.

-----


As for your post above, with LD_LIBRARY_PATH you can redirect almost everything, except:
a) glibc
b) files and libraries that that path hardcoded in them (RPATH)
c) hardcoded data file locations.

cheers

_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2240

PostPosted: Sun 31 Aug 2014, 03:53    Post subject:  

I guess I'll also answer here as I also have a PM in my inbox from matias.

Unioning only works on live CD distros because aufs is not inlcuded in the kernel of any standard (installable) distros. And since uafs is implemented with a kernel source-code patch, it can't be simply added to an existing setup unless you replace the kernel with your own.

Using LD_PRELOAD and PATH(s) is quick and elegant but will not work for things located under /etc or /usr/share -it only lets you point towards libraries and binaries.

So, chroot is the only thing that can be used everywhere and always. Your example with jack should also work, it's simply a matter of making everything needed visible and available under the chroot directory before chrooting in there. You do this with the 'bind' option to mount.

The whole matter boils down to this: there is no way to cleanly and reliably use 'any' app from 'any' distro anywhere else. The whole woof concept is fatally wounded from the start. Even if it was limited to building from a single distro and version, *for* a single distro and version(some puppy), there would still be problems which would require a different solution. There is no magic way to make programs 'just work' outside of the original context.

It is possible to build a sane live distro, but this requires using sane methods, like using the original toolchain to compile any new programs which are not included in the original distro, or any programs which are included but must be re-compiled to alter the options and/or dependencies. The other place where puppy really 'drops the ball', is that, when installing to hard disk, most of the liveCD infrastructure should be replace with their normal counterparts. This means, for example, using real init processes which differ wildly from the processes used by any liveCD.
Back to top
View user's profile Send private message 
matiasbatero


Joined: 11 Oct 2012
Posts: 60
Location: Mar del Plata, Argentina

PostPosted: Sun 31 Aug 2014, 15:34    Post subject:  

jamesbond wrote:
Matias,

The forum has a lousy PM system. I replied to your PM and the reply got stuck in my "outbox" for days. Anyway, I'm publishing my response to the PM here - I hope you don't mind. If you do mind, let me know I'll remove this post.

-----
cheers


Hi jamesbond!, thank you so much for your response. There is no problem for me! Keep your response here! Very Happy (PM's and your posts came at the same time)

Quote:

You have summarised succintly what "woof-next" from Woof-CE does


Wow, i will check this.

Not always possible if you want to avoid chroot. If you want to avoid chroot you can't bring the original distros glibc. This usually is not a problem, but if the packages requires newer glibc than the one installed on the host, it won't work.

LD_PRELOAD, can load a different GLIBC. That can help in some aspects. But, maybe each app have a custom analysis and can be less practical.


Quote:
If you *do* want to compile, then the best way is to install everything to /opt/your-pkgname - including the libraries, and then set LD_LIBRARY_PATH to point to it. This is how libreoffice does it, this is how firefox does it - they have everything including the libraries in their own directory, all executed under a wrapper script. Thus, no chroot or aufs needed.


Yes, compiling, seems to be the more clean method.

But, then you have mentioned this:

Quote:
With LD_LIBRARY_PATH you can redirect almost everything, except:
a) glibc
b) files and libraries that that path hardcoded in them (RPATH)
c) hardcoded data file locations.


How to solve, RPATH: Look this:
http://linux.die.net/man/1/chrpath
https://nixos.org/patchelf.html

User Amigo, says in this post, that LD_PRELOAD/LIBRARY, helps with binary and libs, that is true.. and he says that cannot redirect /usr/share.. so problems can be started for that point.

Quote:
If you want to avoid compiling, then you'll have to go with chroot method. Make the SFS, create aufs layers on top of it, use bind-mounts to bring /tmp, /dev, /proc, /sys and a few other important directories (perhaps /var too) to it, then chroot. It isn't perfect, but it is close.


It seems the best portable way to do this. It isn't perfect because, this solution eats more space, i think.


Your answers have a good information for me, i will analyze some things and your links, and i will coming with some more questions Smile
Back to top
View user's profile Send private message 
matiasbatero


Joined: 11 Oct 2012
Posts: 60
Location: Mar del Plata, Argentina

PostPosted: Sun 31 Aug 2014, 16:08    Post subject:  

amigo wrote:
I guess I'll also answer here as I also have a PM in my inbox from matias.


Of course!! Your response is welcome! Smile

Quote:
Unioning only works on live CD distros because aufs is not inlcuded in the kernel of any standard (installable) distros. And since uafs is implemented with a kernel source-code patch, it can't be simply added to an existing setup unless you replace the kernel with your own.


wow, i thought that unionfs were available on standard distros.

Quote:
Using LD_PRELOAD and PATH(s) is quick and elegant but will not work for things located under /etc or /usr/share -it only lets you point towards libraries and binaries.


Yes, LD_xxx seems elegant. /usr/share or /etc cannot be redirected. It is a problem. Maybe for some apps, not for all. But, that reason is important, because it not seems to be a general solution.

Quote:
So, chroot is the only thing that can be used everywhere and always. Your example with jack should also work, it's simply a matter of making everything needed visible and available under the chroot directory before chrooting in there. You do this with the 'bind' option to mount.


Yes, i will read some jamesbond links about this. Chroot with mount --bind points, for this moment, is the most portable method.

On puppy linux, chroot can run smoothless because, it never ask for permissions. But, in other distros, chroot approach requires permissions for any app to be executed. Yes, it can be solved. But, the solution needs to be very solid.

Quote:
The whole matter boils down to this: there is no way to cleanly and reliably use 'any' app from 'any' distro anywhere else. The whole woof concept is fatally wounded from the start. Even if it was limited to building from a single distro and version, *for* a single distro and version(some puppy), there would still be problems which would require a different solution. There is no magic way to make programs 'just work' outside of the original context.


I think, that there is no way to make ALL programs working outside of the original context. But, if the solution can be great for 80% is acceptable.
By the way, at this moment, there is no linux distro that can run all available packages from repos. And for such apps, always requires manual intervention to get it running.

Quote:
It is possible to build a sane live distro, but this requires using sane methods, like using the original toolchain to compile any new programs which are not included in the original distro, or any programs which are included but must be re-compiled to alter the options and/or dependencies. The other place where puppy really 'drops the ball', is that, when installing to hard disk, most of the liveCD infrastructure should be replace with their normal counterparts. This means, for example, using real init processes which differ wildly from the processes used by any liveCD.


I agree with you. The big problem on all linux distros, is all management is very centralized. And apps, are built to be friend to that way to do the things. A real solution, is "take the bull by its horns". Build all using sane concepts, solid implementations that can manage the flexibility.
Check: GoboLinux , the team redefines the unix-standard-tree. And look, what are their results. With a re-definition, they can avoid to use package-managers. http://www.gobolinux.org/

Thanks for your answer.
PD: Sorry for my english
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [5 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0971s ][ Queries: 11 (0.0031s) ][ GZIP on ]