The State of Package Management
It doesn't really matter that they are used just a few times. The thing is that *some* program *is* gonna need it -when it needs it. and what do you mean by used? That it actually linked and used by running a program, or that it is a run-time dependency of one, a few or many other progs/libs?
I read your proposal for a package format -Sorry, I don't think it's any better thoought-out than the current one. BTW, the current pet format is already 'pet2' -there was something possibly even more brain-dead in place before... and before that there were *.pup thingies.
I read your proposal for a package format -Sorry, I don't think it's any better thoought-out than the current one. BTW, the current pet format is already 'pet2' -there was something possibly even more brain-dead in place before... and before that there were *.pup thingies.
Would you like to share what you think is a good package format, or how the proposal can be improved?I read your proposal for a package format -Sorry, I don't think it's any better thoought-out than the current one.
I'm pretty sure the current format is only "pet". The only change to the format I know of was the name of the specs file (pkgname-pkgver.specs -> pet.specs), which did not increase the package version.BTW, the current pet format is already 'pet2'
If you`re contemplating statically compiling them, common use is important.
Otherwise it doesn`t mean squat...
It doesn`t provide any solution to the package problem, just insight.
# Package problems:
Different library versions can`t coexist, no version control.
Libraries overwriting or overshadowing other libraries.
Tracking and removal of loose files.
( Any others that I missed? )
Otherwise it doesn`t mean squat...
It doesn`t provide any solution to the package problem, just insight.
# Package problems:
Different library versions can`t coexist, no version control.
Libraries overwriting or overshadowing other libraries.
Tracking and removal of loose files.
( Any others that I missed? )
I don't mean that the current format is called 'pet2', but, as you say, the name of the specs file has changed -and the format of the contents. It's a small change, but even a small change which breaks the 'API' can be significant. src2pkg can create either old-style or new-style pets and of course, that means two separate routines for *that part* of the package-creation code. If ppm (or whatever tool) allows for backward-compatibility, then that is to its' credit -it must have been done deliberately or would not be so.
( Any others that I missed? ) Uh, lots still, but your thinking is certainly much further along than some other folks here.
Sorry, but I'm not really gonna hold any hands here -I've been studying hard for 10 years and it will take anyone else just as long. I won't hold myself back just so that others can catch up. Study other systems and try to figure it all out. Of course debian has the oldest and most elaborate system. It's very hard to beat, but still has a couple of drawbacks. We could all just use someone else's format and tools excpet for the perceived drawbacks. For me, the show-stopper for debian packages/packaging is the reliance on perl and the need for human-intervention in order to create *any* package. the show-stopper for rpm packages is the binary format which means you are alway stuck using some of their tools for installation and creation -even src2pkg uses 'rpmbuild' when creating rpm packages. The rpm 'API' has changed over the years, which just makes the problem worse.
( Any others that I missed? ) Uh, lots still, but your thinking is certainly much further along than some other folks here.
Sorry, but I'm not really gonna hold any hands here -I've been studying hard for 10 years and it will take anyone else just as long. I won't hold myself back just so that others can catch up. Study other systems and try to figure it all out. Of course debian has the oldest and most elaborate system. It's very hard to beat, but still has a couple of drawbacks. We could all just use someone else's format and tools excpet for the perceived drawbacks. For me, the show-stopper for debian packages/packaging is the reliance on perl and the need for human-intervention in order to create *any* package. the show-stopper for rpm packages is the binary format which means you are alway stuck using some of their tools for installation and creation -even src2pkg uses 'rpmbuild' when creating rpm packages. The rpm 'API' has changed over the years, which just makes the problem worse.
What do you think of the pacman system?amigo wrote:Of course debian has the oldest and most elaborate system. It's very hard to beat
I think the real complaint about it from Q5sys earlier was this:
I guess in theory pacman could have a feature that did that somehow, but I it wouldn't really be the 'Arch way'. And the point is that you are expected to read the mailing list to know about major changes that require user intervention or whatever.Those things listed above I wouldnt really consider issues with a package. They are from changes to the core system which require special action above and beyond normal updates. However a user has no idea about them until he gets a failed upgrade... or suddenly finds his system will not load properly after an upgrade. Sometimes I wish pacman had a feature where major changes like that would alert the user before installing.
==================
There were a lot of interesting thoughts in this thread, especially at the beginning. But I can't help thinking:
1) If you're building a distro from scratch, there is no point in reinventing the packaging wheel. A lot of smart people have spent a lot of time solving all the problems of a packaging system. Surely there is at least one mature package management system that pretty much meets your needs. Use it.
Or am I wrong? What is Debian's equivalent for dealing with things like "pacnew" files? Maybe no normal package manager is as "user friendly" for system upgrades as a Puppy live CD or frugal install, which automatically cleans up anything that would mess with the new version...
2) If you're building a distro from another distro's packages (woof) surely it is crazy not to use the other distro's package management system.
3) As Jemimah (I think) said earlier, the package format is not the real problem - the repository is. If this forum is the de facto package manager that is not sustainable. If only one person can put packages in the repository that is not sustainable. I'm not very familiar with the woof based puppies - how do they interact with the repositories of the "parent" distro? Can you just fire up the package manager and install something from them? That's what I thought was the intent, but I couldn't see how to when I had a quick look...
Do you know a good gtkdialog program? Please post a link here
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
amigo
From my perspective you certainly seem the most knowledgeable that I've read, but rather than others proposing ideas which you then say are bad with reasons, couldn't you propose a package management scheme others could maybe implement [or did you, and I missed it?]
Personally, if it improves or modifies Puppy into a better distro, [or even forks like iguleder/sickgut are doing] I would forego backwards compatibility, if it meant packages from some standard repository could be used....or is this just a step too far to still be puppy development...?
At least this discussion seems to be drawing package management too a focus, so that, at least must be good, as long as agreement is reached on not doing things that are causing problems, right?
Aitch
From my perspective you certainly seem the most knowledgeable that I've read, but rather than others proposing ideas which you then say are bad with reasons, couldn't you propose a package management scheme others could maybe implement [or did you, and I missed it?]
Personally, if it improves or modifies Puppy into a better distro, [or even forks like iguleder/sickgut are doing] I would forego backwards compatibility, if it meant packages from some standard repository could be used....or is this just a step too far to still be puppy development...?
At least this discussion seems to be drawing package management too a focus, so that, at least must be good, as long as agreement is reached on not doing things that are causing problems, right?
Aitch
Dont get me wrong I love the pacman system... it is just annoying to have to check the mailing lists for issues before i do an upgrade. And yes, those things only happen during core upgrades. However, since we dont seem to care about doing core upgrades though a package system in puppy (a wise move in my book), that shouldnt be an issue.disciple wrote: What do you think of the pacman system?
I think the real complaint about it from Q5sys earlier was this:I guess in theory pacman could have a feature that did that somehow, but I it wouldn't really be the 'Arch way'. And the point is that you are expected to read the mailing list to know about major changes that require user intervention or whatever.Those things listed above I wouldnt really consider issues with a package. They are from changes to the core system which require special action above and beyond normal updates. However a user has no idea about them until he gets a failed upgrade... or suddenly finds his system will not load properly after an upgrade. Sometimes I wish pacman had a feature where major changes like that would alert the user before installing.
Thats something thats been discussed before. And the issue with that is simply that most of the core devs in the puppylinux community are all working on their own pupplet versions. Asside from whoever is developing the offical release, everyone is off doing their work. Sure everyone helps on the offical releases, but the rest of the time time energy and effort are spent elsewhere. They are building their systems for their goals. This divergant focus is where our problem lies. Since each pupplet is remarkably different than another... compatibility of packages is in question.disciple wrote:3) As Jemimah (I think) said earlier, the package format is not the real problem - the repository is. If this forum is the de facto package manager that is not sustainable. If only one person can put packages in the repository that is not sustainable. I'm not very familiar with the woof based puppies - how do they interact with the repositories of the "parent" distro? Can you just fire up the package manager and install something from them? That's what I thought was the intent, but I couldn't see how to when I had a quick look...
To quote from here
So with that in mind I suggested we only work to build one for the official release. But again we come to the problem of who's going to do the work. With each of the core devs doing their own thing... are they going to be expected to build packages for the official release?jemimah wrote:The problem is, nearly every one of our core devs has their own puplet. So we're all compiling the same packages, and they're generally not compatible.Q5sys wrote:Well cant we start small and only increase in size as we have people on board to assist? We've got a pretty solid core group of developers. Each could work on their own little sliver, that way its not outsourced to random people who really dont care about the work. I dont really want to give them more work... but if they just simply approve or veto stuff... Im sure myself or some other maintainer can do the necessary tweaks here and there.jemimah wrote:Ideally, we could just patch the existing ppm. I'm almost ready to work on something like this.
The hosting part should be fairly simple - just ftp access the way smokey01 has his site setup.
The difficult part is finding reliable people to be repo maintainers.
For instance, in Saluki, I'm trying really hard to categorize things in a way that makes sense (to me anyway). So for each pet, I have to modify the .desktop file to make sure the categories are consistent, it has a full size icon, the name, comment, and generic name are set correctly. So even for stuff where the binaries are fine - I have to rebuilt the pet.
What would be most useful to the users, obviously, is if we could all agree on some sort of standard and have a central repo with packages that work on all puplets (at least the recent ones). This is quite possible for Racy/Wary but if you throw Slacko, Lupu, and Dpup into the mix, I don't think it'll succeed.
I think to start, we really need to figure out who is interested in being a repo maintainer and under what conditions they would be interested contributing.
If we can solve the social problem - I'm sure I can make the PPM support our solution.
Read through that thread to get the jist of the discussion there. No reason for me to quote it all here. While that thread started off about a package site... it gets in the issue of package management a little.
Diagnosing incompatibilities in new package builds can be a real pain.
Just statically compile anything that`s incompatible, conflicts are solved.
Not a perfect solution, but there`s few options with the current setup.
No matter how good Barry makes Puppy, one build can`t fit all needs.
But if a mostly complete set of dependency tree files could be had...
Then anyone trying to compile apps. would have a much easier time.
It`s a fairly easy thing to do ( time consuming ), and gives statistics.
And of course they would require updating as dependencies changed.
Q: Does anyone know how Slackware does tree files or handles deps.?
Just statically compile anything that`s incompatible, conflicts are solved.
Not a perfect solution, but there`s few options with the current setup.
No matter how good Barry makes Puppy, one build can`t fit all needs.
But if a mostly complete set of dependency tree files could be had...
Then anyone trying to compile apps. would have a much easier time.
It`s a fairly easy thing to do ( time consuming ), and gives statistics.
And of course they would require updating as dependencies changed.
Q: Does anyone know how Slackware does tree files or handles deps.?
That is one of the problems with Puppy's structure - everybody works on their own project alone, and doesn't share anything. I've created Ppkg to try to unite some the package creation process, but it still doesn't solve the high number of unmaintained puplets.So with that in mind I suggested we only work to build one for the official release. But again we come to the problem of who's going to do the work. With each of the core devs doing their own thing... are they going to be expected to build packages for the official release?
The "current setup" can be changed to something better - which is a topic of this thread.Diagnosing incompatibilities in new package builds can be a real pain.
Just statically compile anything that`s incompatible, conflicts are solved.
Not a perfect solution, but there`s few options with the current setup.
How does creating a new package format not create further fragmentation?noryb009 wrote: That is one of the problems with Puppy's structure - everybody works on their own project alone, and doesn't share anything. I've created Ppkg to try to unite some the package creation process, but it still doesn't solve the high number of unmaintained puplets.
Fragmentation - "The process or state of breaking or being broken into small or separate parts."How does creating a new package format not create further fragmentation?
Doing anything new creates fragmentation - but some new things are good. Puppy wouldn't be here today without IBM making their own computer, Linus making his own operating system, or Barry making his own distribution. A new package format would be good for Puppy, if it is made by, and supported by, the developers in the community. If everyone supports it (or at least likes it better then the PET format), then people will use it, and there would only be a short period of fragmentation.
Package management
My thoughts on the matter:
For a package management system to really work it will have to be adopted by Mr. Kauler and incorporated into woof. Period full stop.
For example, take Lucidpup 5.1 and look in /root/.packages/builtin_files. Each builtin package has a file listing each file of that package. 433 packages in all. That is OK but we still have no idea how those packages were compiled or what the ownership and permissions were when installed, among other things.
The current packages listed in the ppm 'database' with file information are the builtin files and the user installed packages. What is not listed and (to me) the killer for implementing decent puppy package management is the database info on the 990 packages listed in woof-installed-packages. There isn't any info at all for the files those packages provide. We cannot even know how many of the builtin files were over written by the woof installed packages, if any. The true package info in the as delivered puppy is unknown and unknowable, and will remain so until decent package management is built into woof, with every package and it's files accounted for, including woof-installed-files. Then management could be continued with the ppm. It will never be truly good management as long as compiled packages from various other distros are plugged into puppy, but at least package and related file info could be provided for everything.
So, creating a new pet.spec or adding or changing fields to the pet.spec alone is not going to allow sensible package management. Even a better ppm will not get the job done until it is implemented at the root, in woof. Until then the best we can do is improve the handling of the user installed packages.
For a package management system to really work it will have to be adopted by Mr. Kauler and incorporated into woof. Period full stop.
For example, take Lucidpup 5.1 and look in /root/.packages/builtin_files. Each builtin package has a file listing each file of that package. 433 packages in all. That is OK but we still have no idea how those packages were compiled or what the ownership and permissions were when installed, among other things.
The current packages listed in the ppm 'database' with file information are the builtin files and the user installed packages. What is not listed and (to me) the killer for implementing decent puppy package management is the database info on the 990 packages listed in woof-installed-packages. There isn't any info at all for the files those packages provide. We cannot even know how many of the builtin files were over written by the woof installed packages, if any. The true package info in the as delivered puppy is unknown and unknowable, and will remain so until decent package management is built into woof, with every package and it's files accounted for, including woof-installed-files. Then management could be continued with the ppm. It will never be truly good management as long as compiled packages from various other distros are plugged into puppy, but at least package and related file info could be provided for everything.
So, creating a new pet.spec or adding or changing fields to the pet.spec alone is not going to allow sensible package management. Even a better ppm will not get the job done until it is implemented at the root, in woof. Until then the best we can do is improve the handling of the user installed packages.

This is the only way, unless you fork puppy. I would much rather have a format specification in place before it is brought to Barry, however.For a package management system to really work it will have to be adopted by Mr. Kauler and incorporated into woof.
Anything but the current PPM. It doesn't do any dependency resolution (outside of the official repositories) and doesn't even handle package upgrades or file conflicts.Then management could be continued with the ppm.
Without changing woof, the changes get put into a puplet made by one person which will eventually become unmaintained. This is why I am strongly suggesting the new format be made by the packagers who will be using it, as there is not only a much better chance that Barry will accept it into woof, but a much better chance that the format would be used by packagers.So, creating a new pet.spec or adding or changing fields to the pet.spec alone is not going to allow sensible package management. Even a better ppm will not get the job done until it is implemented at the root, in woof. Until then the best we can do is improve the handling of the user installed packages.
This looks liked good progress -Aitch wrote:At least this discussion seems to be drawing package management to a focus, so that, at least must be good, as long as agreement is reached on not doing things that are causing problems, right?
Question: As devs do you not feel you have a right to ask Barry to help sort out the chaos he's created, or risk good devs forever leaving/forking, in an effort to create a good working, useable and updatable OS
I don't see it as insulting, as even he must realise the problems he's creating for you to try to sort out....he does have a natural responsibility for his creation which you guys don't, after all....?
Else Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version....
Good discussion, guys'n'gals, thanks!
btw, I think jrb's pinstall.sh script idea needs more awareness, so here's a link
http://www.murga-linux.com/puppy/viewto ... 808#603808
Aitch
Barry is very quick to accept bug fixes, and enhancements, but it seems unlikely that he'll want to rewrite woof.
Here's the thing. Package management is highly problematic with a system that specializes in frugal installs. The right way to upgrade or change out packages is to rebuild the whole system - which is not a job for the package manager. The PPM should be simple and help you keep your save file small.
What you should be asking for is a better, simpler, more user-friendly puppy builder.
Here's the thing. Package management is highly problematic with a system that specializes in frugal installs. The right way to upgrade or change out packages is to rebuild the whole system - which is not a job for the package manager. The PPM should be simple and help you keep your save file small.
What you should be asking for is a better, simpler, more user-friendly puppy builder.
jpeps, the light gets brighter the closer you get to the end of the tunnel.
"connect listed builtin files with right pet.spec" This illustrates why the idea of a central database of package specs is un-workable. They require lots of work to compose and are rarely up-to-date -they also have no easy way of reflecting past their status.
"full of debs" -If your gonna have your own distro, surely you should be consistent about the content, where it comes from and how it is constructed -or ??
Aitch, "ask Barry to help sort out the chaos" -surely he would have done this if he thought it right or sensible. After all, it is *he* who would most benefit from it.
"good devs forever leaving/forking" Happens all the time. The best *have* spoken to BK about it without result.
"Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version" That's the most concise description of Puppy I have ever seen. Starts out broken, bugs don't get fixed, not up-gradeable -incapable of learning from its' own mistakes for the future benefit of both devs and users. I am skeptical, though, about the validity of this: "less broken later version"
Aitch, I *do* have a very workable, independent package format in place and working very well. Still undergoing some refinement, but already past the point of needing any changes which would affect backard or forward compatibility. Of course, the package format is supported by tools for the creation of compliant packages (src2pkg and others), and the tools for sanely installing, adding, removing, replacing or upgrading packages. There are also advanced tools which analyze the database created by during the above operations. There is still lots to do there, but new little things get added nearly every day. It certainly is functional enough for managing a completze distro with thousands of packages.
Unfortunately, writing an acceptable document which describes even the most basic concepts or specifications of the package format and tools would take me many, many hours of work. The tools do, of course, include '--help' functionality and man-pages, but they don't tell the whole story. Such documents will exist one day, but I'm still much to busy implementing advanced features (like automatic dependency *resolution*) to work on that now. It would be a waste of time to write a nice document when some aspects of the format and the tools are still changing -even though the changes are minor.
Note, while automatic dependency resolution is still not implement in the installer tools, the format already contains the needed 'slots' for the database files and they are generated automatically by src2pkg when building *.tpkg packages. The packages also already contain these files, as needed.
The 'resolver' itself will not be hugely complex and the algorithms will be completly separate from the normal installation tools. Even the tpkg-upgrade tool does not directly install the new package and remove the old one. It uses the individual tools written for each purpose. The same division of labor will be used for any GUI package management tools -they will rely on the underlying tools. This serves at least two purposes -it keeps our 'meaty' code for actually de-compressing archives, running install scripts and updating the database separate from 'window-dressing' code for a GUI. It also means that the GUI code is cleaner and less complex -read easier-to-write and easier to understand or maintain. A low-level tool for each individual operation, with higher-level tools serving as a front-end to them -the UNIX way, you know.
Since I'm 'rolling my own', I don't have to be concerned about getting anyone's 'approval'. If it works it will get adopted by others. Actually, I entertain no desire to have it used by Puppy, as by association it might be identified with Puppy. In fact, I don't even care if nobody else ever uses my format -I've done it to meet my own needs and for the pleasure of creating something that 'just works'.
"connect listed builtin files with right pet.spec" This illustrates why the idea of a central database of package specs is un-workable. They require lots of work to compose and are rarely up-to-date -they also have no easy way of reflecting past their status.
"full of debs" -If your gonna have your own distro, surely you should be consistent about the content, where it comes from and how it is constructed -or ??
Aitch, "ask Barry to help sort out the chaos" -surely he would have done this if he thought it right or sensible. After all, it is *he* who would most benefit from it.
"good devs forever leaving/forking" Happens all the time. The best *have* spoken to BK about it without result.
"Puppy is reduced to sort of being a throwaway OS which you don't so much update, as replace with a slightly less broken later version" That's the most concise description of Puppy I have ever seen. Starts out broken, bugs don't get fixed, not up-gradeable -incapable of learning from its' own mistakes for the future benefit of both devs and users. I am skeptical, though, about the validity of this: "less broken later version"
Aitch, I *do* have a very workable, independent package format in place and working very well. Still undergoing some refinement, but already past the point of needing any changes which would affect backard or forward compatibility. Of course, the package format is supported by tools for the creation of compliant packages (src2pkg and others), and the tools for sanely installing, adding, removing, replacing or upgrading packages. There are also advanced tools which analyze the database created by during the above operations. There is still lots to do there, but new little things get added nearly every day. It certainly is functional enough for managing a completze distro with thousands of packages.
Unfortunately, writing an acceptable document which describes even the most basic concepts or specifications of the package format and tools would take me many, many hours of work. The tools do, of course, include '--help' functionality and man-pages, but they don't tell the whole story. Such documents will exist one day, but I'm still much to busy implementing advanced features (like automatic dependency *resolution*) to work on that now. It would be a waste of time to write a nice document when some aspects of the format and the tools are still changing -even though the changes are minor.
Note, while automatic dependency resolution is still not implement in the installer tools, the format already contains the needed 'slots' for the database files and they are generated automatically by src2pkg when building *.tpkg packages. The packages also already contain these files, as needed.
The 'resolver' itself will not be hugely complex and the algorithms will be completly separate from the normal installation tools. Even the tpkg-upgrade tool does not directly install the new package and remove the old one. It uses the individual tools written for each purpose. The same division of labor will be used for any GUI package management tools -they will rely on the underlying tools. This serves at least two purposes -it keeps our 'meaty' code for actually de-compressing archives, running install scripts and updating the database separate from 'window-dressing' code for a GUI. It also means that the GUI code is cleaner and less complex -read easier-to-write and easier to understand or maintain. A low-level tool for each individual operation, with higher-level tools serving as a front-end to them -the UNIX way, you know.
Since I'm 'rolling my own', I don't have to be concerned about getting anyone's 'approval'. If it works it will get adopted by others. Actually, I entertain no desire to have it used by Puppy, as by association it might be identified with Puppy. In fact, I don't even care if nobody else ever uses my format -I've done it to meet my own needs and for the pleasure of creating something that 'just works'.
jemimah, I believe that a re-think of what comprises a 'frugal' installation would make package-management still possible. For instance, drop the 'frugal' part and simply install the whole system to a disk-image -no need for a save file, only needs a tiny initrd, and uses no RAM for parts of the system. Simply put, a loop-mounted system-in-file. Only the kernel and small initrd would be separate -just as now. I built just such a distro before -nooby would have loved it!
Even without some of the above, a different tactic with the initrd would make it possible and workable to install packages on a 'frugal' system. Even most LiveCD distros which are based on 'real' distros, do a better job with their intirds. The thing is, that by the time the kernel mounts the 'real' root, the system should appear completely normal -where even the normal init scripts work as expected -only the fstab looks different. And this is true whether or not union mounts are used or some other system of integrating various pieces into what looks like one 'whole'.
Even without some of the above, a different tactic with the initrd would make it possible and workable to install packages on a 'frugal' system. Even most LiveCD distros which are based on 'real' distros, do a better job with their intirds. The thing is, that by the time the kernel mounts the 'real' root, the system should appear completely normal -where even the normal init scripts work as expected -only the fstab looks different. And this is true whether or not union mounts are used or some other system of integrating various pieces into what looks like one 'whole'.