Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 01 Sep 2015, 06:02
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
pkgtool CLI package manager
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 4 [46 Posts]   Goto page: 1, 2, 3, 4 Next
Author Message
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Mon 19 Jun 2006, 15:56    Post subject:  pkgtool CLI package manager  

I have hacked together a basic CLI installer for unleashed packages that I'm calling pkgtool. Still a work together so no 'official' release, but it is now in svn. To pull it out of svn issue the following command:
Code:
svn co http://svn2.cvsdude.com/puppylinux/puppy-scripts/pkgtool/trunk

Here's the rundown of features and limitations.
Implemented:
    installation from a local package
    remote retrieval of a package over the internet
    fetch and install a package
    search for a package by name
    configurable mirror list for repositories

To Do:
    implement uninstall
    expand the search function
    port over to Puppy2 (priority)


The biggest limitation is that it has been written with Puppy-1.x in mind, so there are a few changes I need to make in order to function correctly in Puppy2. This is my highest priority project at the moment, sorry but it had to wait until I released Grafpup-104. I should have those particular bugs worked out later this week. With a little more work this will be a perfect complement to OneBone.

I'm also working on a branch with support for dotpet packages, but since there is no consensus on what a dotpet is this is strictly my own take. When I have completed a little more work on that I will release a spec, or alternatively if someone would like to collaborate on it that branch is also in svn under branches instead of trunk. I'll update the README to be a little more informative later this week.

The entire concept is to detach the actual install/uninstall concept from the gui and make it all more flexible. This way other programs can interact with it, and there is a lot more flexibility in gui design. Plus I believe the code will be a bit easier to maintain and/or modify.

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Thu 22 Jun 2006, 01:54    Post subject:  

I just commited a new revision to pkgtool, which should give no problems with Puppy2. It turns out that there was very little that needed changed. After comparing the new Pupget with the old Pupget i was able to see where things were different and make a few tweaks to match. Actually, most of my install function is word for word what Barry has in Pupget. There's just a few differences in the registration section, which I coded myself.

I still have not had time to add the uninstall function, and the search function is not complete to my satisfaction yet either. Currently it will search only through the pacakge names, but what I intend is to have it search that first and then search the package descriptions, and then finally check the built in dependency info field. So in effect you will get a list that goes from most relevent to least, based on the criteria you feed it. I also need to recode it a bit to search any alien packages that are installed and the packages already in the live cd as well.

I've been thinking about gui design a bit now. I think it would be nice to change the install icon on the desktop so that you can just drag a package to it to install, kind of how Barry has the Pupzip icon set up. Other than that i think the two pane window should still stay pretty much untouched, but the frequent use of xmessage is pretty ugly and tired looking. Maybe just use Xdialog throughout for consistency? Or maybe this would be a good opportunity to use the GTK functions in bashdiff. Any thoughts appreciated. That is if anyone is even looking, so far no feedback whatsoever.

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
Lobster
Official Crustacean


Joined: 04 May 2005
Posts: 15154
Location: Paradox Realm

PostPosted: Thu 22 Jun 2006, 02:52    Post subject:  


What you are doing is very worthwhile . . . so really want to encourage what I believe is the most important Puppy project at the moment.

I have posted (which you will have seen) my GUI suggestion
The two panel approach that pupget uses is combined into one. When you tick a piece of software a dialogue 'Install software? / Install all selected software?' would come up.

Extra info would be revealed as a package was scrolled onto . . .

Uninstall? The tick is removed.


http://puppylinux.org/wikka/DotPet

Suggestions needed. Feedback? Comments? This is our next generation Puppy installer - with bells on . . .

_________________
Puppy WIKI

Last edited by Lobster on Thu 22 Jun 2006, 05:33; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Thu 22 Jun 2006, 04:17    Post subject:  

That's a TK gui if I'm not mistaken, right? It's a bit prettier thatn what we have currently, to be sure. I see a lot of work ahead, so let me explain a few finer points.

What we have in Puppy for a package database consists of three textfiles. If you look at packages.tx, livepackages.txt, and alienpackages.txt they are all in a like format with four fields of information. The first field is simply the package name, absolutely essential obviously. The second field is the description, and is exactly what appears in Pupget. Third field is either just the word on or off, depending on whether the package is installed or not. Now the last one contains a bit of categorization, soetimes the size, and sometimes a list of dependencies.

The categories are pretty meaningless as far as designing a gui is concerned, actually. Packages are broken down into CONSAPPS, CONSCORE, GTK1APPS, GTK1CORE, XLIBAPPS, XLIBCORE, GTK2APPS, GTK2CORE, and finally MMAPPS and MMCORE. Oh and also QT3APPS and QT3CORE. Barry did it this way so that when an iso is being built it will warn you if you are missing something from one of the CORE groups in some category, for instance if you choose Opera as the browser it will tell you that you need to also choose QT. Pupget can also use this information, and by dividing up the packages into rough groups like this we have some very basic dependency tracking. very course, of course.

So the field for categorization already has a valid use and can't be hijacked for the purpose of recategorizing the applications into something more human understandable, like graphics, office, and network. This makes dividing into categories a bit more difficult. Not impossible, mind you, just a bit more difficult.

I can see an out, but it might require Barry's direct involvement. He already has that fourth field subdivided into three relevent sections, or subfields. We could add another I guess, for categorization by purpose. This would not mess up Pupget the way it currently works, or unleashed, except that that info would display when you hover the mouse cursor over a package since that's the field that Xdialog uses as a tooltip.

Personally I rather like the two pane window, to tell you the truth. But breaking it down into categories, by purpose, would be good. There would of course have to be a misc section for any alien packages, and any that just defy categorization or don't fit nicely into a category. This type of thing would have to be done in something other than Xdialog, of course. Which means parsing the lists and rewriting them in some new form, because right now those files are formatted to be fed right into Xdialog. I'm a bit against doing it this way, because then it slows the program down and people already have complained that pupget runs too slow. A better solution would be to just change the lists into another form entirely, readable by whatever gui is being used. But then that messes up unleashed. Very circular, huh?

If we came up with a great gui, that used a different format for packages.txt, maybe the createpuppy script in unleashed could be modified to parse it into the correct form for our hypothetical gui, so it's done and ready to go in the iso and that part doesn't have to happen when running the package manager. Again, direct involvement from Barry who would have to accept the changes in the unleashed scripts.

In a way I guess this is a rant. I just want you to know how much work you seem to asking that we take on, considering my main intention was at present a CLI interface to install unleashed packages. Adding the categories goes right to the basic design of the program, and we may as well start from scratch if we're going to do that.

I'd have to summarize right now and say, lets take some baby steps here before we go that far. Separating the gui from the installation functions is a good first step in my opinion and will have an immediate effect on the system's flexibility, which we can then build on as we go.

Here's the major things that can be done right now without redesigning the whole thing from scratch. We can get click to install prett easily from this point. Klh's downloader can use this to install unleashed packages right alongside dotpups, which demonstrates one way of making package installation more flexible. We can now have a package manager in OneBone, after I write the uninstall function. I can use what is already in the livepackages.txt file to implement dependency resolution, although it will take a bit of tinkering to get that to work for alien packages since they have no entry in livepackages.txt. Let me get that much in order and then after that we can work on adding other functions.

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
Lobster
Official Crustacean


Joined: 04 May 2005
Posts: 15154
Location: Paradox Realm

PostPosted: Thu 22 Jun 2006, 06:22    Post subject:  

Quote:
That's a TK gui if I'm not mistaken, right? It's a bit prettier that what we have currently, to be sure.


Smile I did it in XaraLX - good I am glad you feel it can be done in Tk. Tkl Visual designer works in Puppy.

Thanks for all your advisement. Let me start with a story. Kanotix had a one-klik installer (it also works with Debian) at the time I tested it - only 10% of the one kliks worked. Boring - there was only about 20 - so that is 2
Now then 90% of our users only require 10 or 20 extra packages . . . we can work towards that as our first module.

So maybe we should focus on combining pupgets and dotpups and make sure the .pet works for GuestToos .pups which are the most stable? You are looking towards everything. I am looking towards stages. I want rarsa involved with Subversion, ideally I would like Kenny as project leader. Mu has expressed interest and I would value more input. Barry I am sure is watching the thread. I think this should be our second community led project and so Barry is welcome to take any role he wishes.

Quote:
This makes dividing into categories a bit more difficult. Not impossible, mind you, just a bit more difficult.


I am kind of end user motivated. Are categories essential?
well with 10-50 packages not really. Evetually it will be required - don't you think? I am just glad .pet is on its way.
The process for installing an unleashed package at the moment is arduous . . .
Find the page - that can take 15 minutes, download and work out it is installed as an alien package that can take a couple of hours to remember - many will never get it or bother . . .
Pupget is slow but it does install and uninstall rather neatly . . .
You seem to be working towards the priorities . . . I will try creating a two box and one box GUI in Tk for testing puposes. Others might try too . . .
There should be a link on the wiki pages on tkl - my next stop . . .

Smile

"Input! More Inpupt!"

_________________
Puppy WIKI
Back to top
View user's profile Send private message Visit poster's website 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7455
Location: Perth, Western Australia

PostPosted: Thu 22 Jun 2006, 10:08    Post subject:  

Nathan,
Don't use bash-diff, as I don't know if it will stay in puppy. I think latest busybox
has bash, and may go for that, for the small size.

I have specified Perl as PERLCORE, and GPRename (a perl/gtk application)
as PERLAPPS, but this is not so good.
Might be better to have the Perl pkg as CONSCORE, since it looks like it is
going to be a fixture in Puppy. Then, GPRename can be GTKAPPS. ...sensible,
I think.
Back to top
View user's profile Send private message Visit poster's website 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Thu 22 Jun 2006, 12:02    Post subject:  

Quote:
I did it in XaraLX - good I am glad you feel it can be done in Tk.

HaHa, I knew there was a catch.
Quote:
Now then 90% of our users only require 10 or 20 extra packages . . . we can work towards that as our first module.

Okay, this is something that could be done now probably. Your top twenty are likely to be in one of three formats, either a dotpup, a pupget, or a squashfile. It's pretty easy to have a script install a dotpup, it's pretty easy to do that with an unleashed package using pkgtool. Someone would then want to write a dialog that downloads a squashfile to /mnt/home, or better yet dev_save. But be careful, it should check to be sure it's not writing over an existing squashfile first before starting the download. Pretty simple overall, and the gui could take a lot of differrent forms.
Quote:
I think this should be our second community led project

I mentioned as much when we were working on 109CE, that I thought package management would be a good thing to work on together. That's why I wanted to put his into subversion, too.
Quote:
I am kind of end user motivated. Are categories essential?
well with 10-50 packages not really. Evetually it will be required - don't you think?

I wholly agree with you on that, I'm just not ready to go there yet. We really should do it eventually, though.

For a tick box type of gui GTK might be a better match, since it already has a fairly easy to use radiolist widget. I really know nothing of how to program in TK, but it would be a good match since it's definately in Puppy long term.
Quote:
Don't use bash-diff, as I don't know if it will stay in puppy. I think latest busybox
has bash, and may go for that, for the small size.

Okay, no bash-diff. And I was kind of looking forward to trying it out. Very interesting news about busybox...

I would tend to agree with you about making perl CONSCORE, and GPRename GTKAPPS, or probably GTK2APPS right? Haven't looked at it yet, don't know. I would suggest that for the next release make sure there is as much dependency info in the fourth field of packages.txt as possible, in case someone creates a more basic version of Puppy that leaves out something like perl. I think at one point you must have intended to use that dependency info but never got around to it yet.

For dependency info I think a better way to do it would be to place a textfile in the top level of each package with a list of it's dependencies. That way it's not hard coded into Puppy when the iso is created in unleashed, and can be used by alien packages also. A small tweak to the format, that could be easily dealt with in Pupget. If it was not to be used yet just have Pupget delete the file, or move it to .packages like everything else. it would also require a small change in unleashed to deal with. With well over 300 unleashed packages I realize this would have to be phased in slowly.

So as example, for GPRename there would be a textfile, I'll call it required.txt, inside the top level of the package. It woul list perl and gtk (possibly a few others as well), while gtk would probably mention X. So it could be cascading, in other words with a full implementation someone could download Onebone and install to HD, and then by installing GPRename (or some other graphical utility) the package manager would go through the list of deps and get you a full working X environment. For now just a few packages could have this little change, but eventually it could be done for all.

Any opinion on this, Barry? I try to avoid making changes to the package format itself, so far as possible, but I think this little change would be a big plus. And I've never been thrilled about how everything is coded right into package.txt and remains rigid until the next iso released.

Speaking of which one of the projects on the back burner would allow updating those lists Wink

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7455
Location: Perth, Western Australia

PostPosted: Thu 22 Jun 2006, 22:43    Post subject:  

yes, a dependencies textfile seems like a good idea.
Back to top
View user's profile Send private message Visit poster's website 
raffy

Joined: 25 May 2005
Posts: 4827
Location: Manila

PostPosted: Fri 23 Jun 2006, 01:20    Post subject: Priority  

Yes, it's good you're working on this, Nathan.

The discussion is quite long on GUI already, so let me just focus on the CLI.

Maybe it can help to include categories in file names, just the way X tools are named beginning with X.

An easy way to install packages in CLI is perhaps to tell the user to download the chosen packages to a folder, say "install_me" then run the installer. Dependencies can then be checked and the user told to get the files that are still lacking. Dotpups have a way of checking the md5sum of downloaded files, so that is useful, too. Successfully installed packages get transferred to another folder, "OK2delete".

I've had a number of failed pupgets lately (often a mismatch of file name from the one listed)*, and I guess the user can be given more room to do things in the tradition of the dotpups (choose, download, install). Only this time, it's in CLI. Smile

Just my 0.0002 cents Embarassed

* This is largely understandable because of the huge work demanded in transitioning to Puppy2.
Back to top
View user's profile Send private message 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Fri 23 Jun 2006, 01:44    Post subject:  

I'm working a bit on the uninstall function, and the upgrades to the search function I mentioned. It might take a little longer to get done than usual because of the long commute I'm running the next few days. I want to make sure I've got it essentially working again before I check back in to svn, haha I managed to break it pretty bad today.

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Tue 27 Jun 2006, 19:50    Post subject:  

Okay, the search upgrades are mostly done in the latest commit. I also improved the get_install function somewhat for better reliablility, but it still misses it's target occasionally. For instance typing 'pkgtool -getinstall samba' will try to install gtksamba, since that is the first entry grep finds that contians the word "samba". So I need to get that fixed up a bit, need to code a way to get exactly the right package.

I also coded a preliminary dependency resolution function, which reads the + signs in livepackages.txt to determine which package is needed. It works most of the timeand is actually pretty slick, but since it depends on the get_install function if that misbehaves it might get the wrong package by mistake. The get_install function does tell you which package you are about to install and lets you back out if it's wrong, so no harm done if it gets the wrong package at this point.

I'm thinking, I'd like to keep this in even if we do add the dependency file to each package like I mentioned above, because it's nicer to know which packages are going to be installed before downloading the first one. The textfile inside each package with a list of deps would get anything that was missed, and allow alien packages to use the system also.

I still haven't coded that darned uninstall function yet, will have to get moving.

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Wed 28 Jun 2006, 18:08    Post subject:  

Well I finally got around to coding the uninstall function for this, but right now it should be considered beta at best. It seams to be successfully uninstalling without messing anything up, fixes the entries in livepackages.txt and alienpackages.txt right, but for whatever reason it is not updating the keywords correctly. So menu entries remain as well as the help index entry. I will have to fix that but have already done enough today.

There was also another issue I hadn't expected which is now taken care of. The file livepackages.txt contains a list of all of the packages that are available for installation, but it does not exist until the first time you run pupget apparently. I have added code to pkgtool to fix that issue, which should properly generate that file if it does not yet exist.

I've been in OneBone all morning and I'm loving it. For kicks I wanted to see how long it would take to get a basic working X environment using pkgtool, and surprisingly the answer was about five minutes and only about five or six packages to install. I'll probably remove them now, of course, but the thing runs surprisingly fast with only basic xlib and gtk1 applications. Of course even faster in text mode.

I already commited the latest changes in pkgtool to svn, so it's there if anyone is interested. I'm getting pretty close to what I think would be a good beta release but need to do a couple more things first.

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
BarryK
Puppy Master


Joined: 09 May 2005
Posts: 7455
Location: Perth, Western Australia

PostPosted: Wed 28 Jun 2006, 23:21    Post subject:  

Nathan, if you're really comfortable using bash script, and most keen on keeping bash-diff, I will consider keeping it. One thing, does it's GTK functionality offer anything better than Xdialog and Gtkdialog that we already have?
Back to top
View user's profile Send private message Visit poster's website 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Wed 28 Jun 2006, 23:32    Post subject:  

To be perfectly honest I haven't done more than just a little experimenting with it. I don't think the gtk functions alone are a reason to keep it but I'll point out that the xml functions are probably pretty worthwhile, this could be put to very good use with more applications going to xml for data storage all of the time. Also, with the gtk functions it seems a little more straightforward with bashdiff than with gtkdialog, in other words easier and more accessible.

As stated though, I'm not an expert on this. It might be better to put the same question to Rarsa.

So I'm wondering now, just how good is the new busybox bash? There were always a few inconsistencies with busybox ash so I'm hoping we don't run into any strange behavior if the plan is to migrate to that.

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
Nathan F


Joined: 08 Jun 2005
Posts: 1760
Location: Wadsworth, OH (occasionally home)

PostPosted: Thu 29 Jun 2006, 13:48    Post subject:  

Well I added a simple fix that makes the get_install function a little more reliable, which should improve the reliability of the dependency resolution function also. Previously I used this bit of code to sort the input and try to get the right package:
Code:
cat /root/.packages/livepackages.txt | cut -f 1 -d ' ' | grep "$PKG" | head -n 1 | sed -e's/\"//g'

Which worked most of the time, but feed that bit "samba" as a query, and it will return gtksamba, since that is the first entry found. The fix, search for "samba instead of just samba, code:
Code:
cat /root/.packages/livepackages.txt | cut -f 1 -d ' ' | grep "\"$PKG" | head -n 1 | sed -e's/\"//g'

There's still an issue with the dependency tracking, which might appear occasionally. Many of the entries have at the end something like +package, for instance, which is saying basically that this one depends on 'package' but is not specific to the version number. My little fix will stop it from grabbing, say, 'apackage' by mistake. But it might still try to get 'package_a', for instance. I thought maybe I could go for something like, grep for "package-, which would look for the package with a dash after it, but that would fail in the cases where the version number was given. I haven't scanned packages.txt closely enough to know how often one might run into this type of issue, but I know I ran into it at least once during testing.

Hey, wait, I think the 'head -n 1' should take care of that after all. Maybe not an issue now, I'll have to do some more testing.

I also found a discrepency that needs resolved in packages.txt. The entry for esvn has +svn at it's tail, but the svn client package is named subversion_client. So no way for any script to figure it out unless you just make a special case for the +svn entry. It would be much better form to have the entry read +subversion_client. In fact it would be nice to go through the entire file and fix up this section a little bit, to make sure it's up to date and has enough information to be useful.

Another issue comes when there are multiple dependencies listed, the script just conks out on me right now and does not process any of the deps. My own code is probably at fault here so I will have to have another look at it.

Well I've said from the start that this should be usable with a gui, and here's how that should work. The script overall accepts a few types of arguments. The syntax is
Code:
pkgtool (option) (package)

It can also accept a third option, which should be hidden from end users, which is the gui key. So if you were to pass this command
Code:
pkgtool -getinstall qt pupget_get_install

It would bypass any interactive portions and run silently, returning an exit code at the end. This allows for anything interactive to be in the gui script, so a console does not have to open for a person to type in yes or no. I still have to code this into a couple of the functions but that just might be the last thing that has to be done for the time being.

What this boils down to for someone wanting to create a gui wuold be this, you must take care of anything requiring the users input within the gui. For instance, pkgtool can download and install the package even without the version number, but asks you first if it has the right one. From a gui you would pass pkgtool the packagename and version number, plus the gui key, and it then runs silently without user input. I think that's a pretty good way to separate the gui from the functions while still being interractive in both CLI and gui. If anyone knows a better way please pass on the knowledge.

Note that the search function is a special case, as the third option is used by the end user. For instance
Code:
pkgtool -search on

returns a list of packages you have installed, including any alien packages that registered with puget
Code:
pkgtool -search on -v

expands the search to include all of the packages that are on the live cd as well. I intend to put a divider in the listing to show where those begin, but have not done so yet.

I was also thinking, with the package lists in their present form the info should be readable by not only Xdialog but also Dialog, so it might be worthwhile to either develop a Dialog based interface for command line usage or go a step further and make Pupget into just a wrapper that can determine if X is running and launch either a graphical or CLI interface if appropriate.

Anyway, probably just a few more days tinkering and I will put out something that should work without any hassles. I'm thinking I will release it as a dotpup for regular Puppy use and as a tarball with an installation script for OneBone use. Still welcoming any input.

Nathan

_________________
Bring on the locusts ...
Back to top
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger 
Display posts from previous:   Sort by:   
Page 1 of 4 [46 Posts]   Goto page: 1, 2, 3, 4 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1016s ][ Queries: 12 (0.0193s) ][ GZIP on ]