No more dependency management

Under development: PCMCIA, wireless, etc.
Post Reply

Which distro to test?

Debian Wheezy
1
10%
Debian Jessie
1
10%
Slackware
7
70%
PclinuxOS
0
No votes
Ubuntu LTS
1
10%
Mint
0
No votes
Other
0
No votes
 
Total votes: 10

Message
Author
User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

No more dependency management

#1 Post by technosaurus »

It is possible to eliminate dependency management altogether by just fetching the appropriate file on demand from a server with all kernel modules and packages installed.

This requires a basic system with network access, patches to a few utils and wrappers around some syscalls.

insmod and modprobe would need to be patched to try to download the module if a local one did not exist

the open() syscall would need to be wrapped to try to fetch files on local fail
as would execve() and possibly access()

A basic system could start with the minimal requirements to access the net, a directory full of every .desktop file in the distro and an xdg-open style program to open files based on MimeType.

Once you have setup your network and want to open google.com by entering xdg-open http://www.google.com (puppy's equivalent to xdg-open is Rox-Filer btw)
This will give a menu derived from .desktop files that can open a url.
Once you select it, it is moved to /usr/share/applications
Then it tries to run the Exec= command, but we never actually installed it
The execve wrapper will capture this and download the binary
But the binary may need shared libraries to run...
These are captured by a modified dlopen to download those.
But other files may be needed (localization for instance)
These are captured by a wrapper around the open() syscall.
Sometimes they may need a certain kernel module though.
These can downloaded within modprobe/insmod if no local module is found.

Any thoughts?
Last edited by technosaurus on Sat 03 Jan 2015, 01:46, edited 1 time in total.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#2 Post by amigo »

And so, how do you decide which is the appropriate file to download? What you propose is simply moving dependency resolution into glibc or the kernel itself -neither of which should be worrying about such things. Someone still has to generate the information and make it available. Or should glibc crawl the entire internet looking for a certain *.so file, or maybe have the crawlers results hard-coded into glibc? How would I tell it to use a *certain* object file which has been prepared for the purpose? How would it know that the 'man' program actually needs 'groff' in order work? How would it determine which language I want to use? Where would the URL's be stored where open() should look for files? Would I be able to specify which network device to use? What about if the network is not yet configured? Should the kernel use some arbitrary URL after self-configuring the device?

The only way to completely avoid dependency management is to have all dependencies completely installed with a base installation. Slackware does it this way -until you need something whose required libs are not part of a complete slackware installation. android avoids the problem by having a single software platform (java) which is able to do any/everything.

goingnuts
Posts: 932
Joined: Sun 07 Dec 2008, 13:33
Contact:

#3 Post by goingnuts »

I thought that static linking of all software and packages solved the dependency problem and even broadened the usage across the various distros. Different architectures still is a challenge - but for me the solution seems to be single independent packages with as broad a usage as possible.

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#4 Post by greengeek »

The server would surely have to be specific to only one distro and version would it not? Otherwise how would it differentiate between requests for a specific Lucid puppy lib that had the same name as a Slacko puppy lib?

Each request would have to be prefixed with the name and version of the requesting operating system (as a minimum...).

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#5 Post by technosaurus »

amigo wrote:And so, how do you decide which is the appropriate file to download? What you propose is simply moving dependency resolution into glibc ....
Same style that puppy already uses to select the fastest mirror.
Only the c library would need to be modified and only a few calls.
Take the example of man and groff.
When man is executed the execve fails at first, downloads it to /usr/bin (only 1 directory for binaries) and tries again successfully.
At some time during the execution the process is repeated for groff when man either directly or indirectly calls execve.


The process is similar for libraries opened with dlopen or files opened with open except you just keep the path (resolved to an absolute path). If no path is given then libs will come from /usr/lib (lib32/lib64 will be symlinks - and no plan for lib32 on a 64 bit distro)

I did some experimenting with putting all of the kernel modules in a single directory before and got significant load improvement from having a much less complicated modules.dep (and friends) as well as reduced inodes and directory walking... It turns out that it is ideal for this situation. When module are probed/loaded it fetches /lib/modules/version/module in a similar way.

@goingnuts - licensing issues aside, static binaries still don't include assets like localization.

@greengeek you could have one server hosting multiple distros though as wheezy.example.com and tahr.example.com.

Setting the server up is as simple as installing all packages (that don't have conflicts ) move all the bins to /usr/bin, libs to /usr/lib, modules to /lib/modules/kernel-ver. It can be any https server that can host static files... Maybe with rsync on the side to easily do updates.

Since you know what the other side has, all files could also be precompressed.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#6 Post by technosaurus »

I'm going to test this out starting next week with coding. I'm planning to use Debian or slackware because it's easy to dump their packages to a directory and the source and patches are readily available, but I put up a poll if anyone has strong arguments for another distro.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
smokey01
Posts: 2813
Joined: Sat 30 Dec 2006, 23:15
Location: South Australia :-(
Contact:

#7 Post by smokey01 »

The poll doesn't appear to be working.

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#8 Post by technosaurus »

weird, it just worked for me ... maybe I needed to vote 1st
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
smokey01
Posts: 2813
Joined: Sat 30 Dec 2006, 23:15
Location: South Australia :-(
Contact:

#9 Post by smokey01 »

Duh, I wasn't logged in.

oui

#10 Post by oui »

Hi

I did select Slackware of course but really regret at the same time that the Puppy world does not will to take LinuxFromScratch under consideration and continues to do so :roll:

Kind rgds

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#11 Post by technosaurus »

Wow! so far Slackware has an overwhelming lead. 100% aside from my vote for Debian Wheezy... I should probably add Arch too. At first I was going to vote Slackware but it just somehow seemed wrong to do something so impure to such a purist distro. Now to figure out an easy way to download all their packages. Let me know of any pseudo-official repos that I should add for additional packages; IIRC there are a few for Slackware since their official package list is dwarfed by Debian (my other reason for choosing Wheezy ... skipped over Jessie due to systemd).
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
James C
Posts: 6618
Joined: Thu 26 Mar 2009, 05:12
Location: Kentucky

#12 Post by James C »

Voted for Slackware......no systemd crap and Wheezy is getting a bit dated.

User avatar
smokey01
Posts: 2813
Joined: Sat 30 Dec 2006, 23:15
Location: South Australia :-(
Contact:

#13 Post by smokey01 »

My reason for voting Slackware is the slackbuilds system. It makes compiling quite simple.

These are good springboards for slackware.
http://slackware.ponce.cc/blog/category ... uilds-org/

http://slackbuilds.org/

http://www.slackware.com/~alien/slackbuilds/

http://www.slackware.com/~alien/

Here is a zip with over 2000 slackbuilds. The script downloads the source.
https://github.com/PhantomX/slackbuilds ... master.zip
or
https://github.com/PhantomX/slackbuilds
Last edited by smokey01 on Sun 04 Jan 2015, 12:42, edited 4 times in total.

User avatar
Ted Dog
Posts: 3965
Joined: Wed 14 Sep 2005, 02:35
Location: Heart of Texas

#14 Post by Ted Dog »

maybe compile on pull like tccboot method from 2005 :x

User avatar
smokey01
Posts: 2813
Joined: Sat 30 Dec 2006, 23:15
Location: South Australia :-(
Contact:

#15 Post by smokey01 »

Techno are you considering gtk3?

Many of the latest versions of software need it.

oui

#16 Post by oui »

LFS also offers an ideal compiling environment. divers LFS derivates offer self compilation and both books LFS and BLFS all instructions to do that step by step!

the problem of Slackware is the same as from Puppy: no central depository and a terrible searching if you want more than usual package :wink: but that will probably not afraid Puppy users :lol: !

(see for the perfect SliTaz organisation: www.slitaz.org, doc.slitaz.org, mirror.slitaz.org, forum.slitaz.org etc.! and a automatic warning system in the doc if a language is older that the mother version of the text!)

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#17 Post by amigo »

Don't we already have PXE for booting from network? If the idea is that you really want to need a partial runtime environment available locally to boot from and then access extras with your method, then where do you draw the line between what is required locally and what is extra (only available with network running and enabled. Another variant which does something similar is httpfs which mounts directories which are online from an already-running system.

Post Reply