Petch-0.3.pet -- CLI Puppy repository search and install

Under development: PCMCIA, wireless, etc.
Message
Author
Caneri
Posts: 1513
Joined: Tue 04 Sep 2007, 13:23
Location: Canada

#16 Post by Caneri »

Hi Tom,

I wonder if adding puppyfiles.ca to the search in Petch would be helpful as well.

MU mirrors his dotpups.de there and if he is down then puppyfiles.ca is available. There are instructions on how to use puppyfiles.ca on the front page there from MU.

I'm not sure if there is a package list for puppyflies.ca though, but there must be as PSI can use it...Maybe Mark can answer that one.

Best,
Eric
[color=darkred][i]Be not afraid to grow slowly, only be afraid of standing still.[/i]
Chinese Proverb[/color]

User avatar
MU
Posts: 13649
Joined: Wed 24 Aug 2005, 16:52
Location: Karlsruhe, Germany
Contact:

#17 Post by MU »

these folders have dotpups.txt, also in small .bz2 format:
http://puppyfiles.ca/dotpupsde/dotpups/
http://puppyfiles.ca/dotpupsde/puppy3/dotpups/

I just see, this one has none yet, I will add it this week:
http://puppyfiles.ca/dotpupsde/puppy4/dotpups/

Mark
[url=http://murga-linux.com/puppy/viewtopic.php?p=173456#173456]my recommended links[/url]

User avatar
tombh
Posts: 422
Joined: Fri 12 Jan 2007, 12:27
Location: Bristol, UK
Contact:

#18 Post by tombh »

@ndujoe1: Didn't see yer post there before, thanks!

@Eric: Where should Petch live on .ca then? Puppyfiles.ca is already included, that's what I really mean when I say dotpups.de as I assumed that the former is a mirror of the latter.

@MU: Thanks, I already had http://puppyfiles.ca/dotpupsde/dotpups/ but not
http://puppyfiles.ca/dotpupsde/puppy3/dotpups/ so I added it.
[url=http://www.tombh.co.uk/colour_memories/contribute.php]Colour-Memories Database[/url]

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#19 Post by Pizzasgood »

Cool. This would be even better if we set up that attachment list, because then it could grab stuff from the forum too. What would be ideal in that respect is just a single .php page that ran an sql query for all .pet, .pup, etc. files (so we don't get things like conky.jpg screenshots).

I happen to have a (disabled) phpBB forum on my site, so I'll see about throwing something like that together this weekend. If I can get something more or less working on my end I can email it to John and see if he'll upload it to the server.
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#20 Post by cb88 »

thanks tombh this is a great development
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#21 Post by Lobster »

Works OK in Muppy
Should be in Puppy - is it too late for 4.1? :cry:

Certainly should be in 4.2 8)

Brilliant!
This could be called from the package manager . . .

Code: Select all

# petch xara
searching ...

       SOURCE        | DIRECTORY      | FILENAME (size)
[ 1] - puppylinux.ca | tpp            | Xara-XtremeV0.7r1785.pet (27447892)
Please type the number of the package you would like to fetch: 
Tom before I forget
Don't think I can meet up in central London this weekend
but will see how it goes . . .
Hope you have a great trip
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

User avatar
tombh
Posts: 422
Joined: Fri 12 Jan 2007, 12:27
Location: Bristol, UK
Contact:

#22 Post by tombh »

@Pizzasgood: Yes it would be even better wouldn't it. I imagine phpBB stores all its attachments in a sub-folder somewhere -- so if worse comes to worse I guess at the least we could just have a PHP script (without any SQL calls) make a daily list of, say all .pets, below a certain directory.

So for anyone that's interested I've attached the script that MU wrote to do this on dotpups.de and how Caneri now does it on puppylinux.ca too.

@cb88: Thanks.

@Lobster: I don't know if Barry will want it in his Puppy because it doesn't just search the official ibiblio repository. But he's very welcome if he does though!
No worries about meeting up :)
Attachments
cronpets.php.tar.gz
(1.02 KiB) Downloaded 601 times
[url=http://www.tombh.co.uk/colour_memories/contribute.php]Colour-Memories Database[/url]

User avatar
HairyWill
Posts: 2928
Joined: Fri 26 May 2006, 23:29
Location: Southampton, UK

#23 Post by HairyWill »

tom
how about getting petch to look for updated packages

for each installed package search for versions with a higher version number

If you call petch on an already installed package it could report the current installed version number.
Will
contribute: [url=http://www.puppylinux.org]community website[/url], [url=http://tinyurl.com/6c3nm6]screenshots[/url], [url=http://tinyurl.com/6j2gbz]puplets[/url], [url=http://tinyurl.com/57gykn]wiki[/url], [url=http://tinyurl.com/5dgr83]rss[/url]

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#24 Post by Pizzasgood »

I looked at that file and dug up the one it creates at puppylinux.ca. Is there any particular reason for the second path entry for each item?


The forums attachments are all stored in the files/ subdirectory, but there's an index.php file there to block a file list from being generated by the server. So we could use something like what you posted, except that it wouldn't filter out packages that are inside a PM. I doubt that anybody would put something confidential inside a .pet package, but you never know. A more likely concern is if they put something inside a .tar.gz file, which I currently have this set to grab.

So I went ahead and wrote something that runs a couple simple queries so it can filter out attachments. Just making something to output a list is too simple, so I added a second mode to it. You can visit it with no options and have a more detailed list, with the file creation time, the attachment description, a link to the post it was posted in, the username of the person who uploaded it, and a link to that user's profile. That list also shows the package's original name, before the forum adjusts things to avoid overwriting existing packages.

The basic mode is accessed by providing the ?view=basic option in the URL. Then it outputs plain-text (set the header to text/plain so the browser treats it like a .txt file) formatted just like the dotpups.txt file on Caneri's server.

One thing to note is that the forum does rename a package after you upload it if the name would conflict with an existing package. It stores the original name, and under normal circumstances I suppose it copies the file to a temporary directory and renames it to that before letting you download it. Nether modes of this script bother with that. The fancy view will display the original name in the link, but the downloaded file uses the actual name the forum named it. The basic view only shows the actual name, since AFAIK I can't have it show the real name without messing up the format (unless this is the function of that duplicate path, which is the main reason I asked about it).


Right now it will grab any file with the following extensions, except those that are inside a PM: 'pet', 'pup', 'zip', 'tar', 'gz', 'rar', 'rpm', 'deb', 'tgz', 'tbz', 'bz2'


I don't have a demo up yet, because it turned out I needed to install the modifications that this forum is using, and I was too lazy to mess around with doing that remotely with my website. I do have a local server running that I tested it on, so I can attach some screenshots along with the actual script. I'll try to rig up a demo later tonight, or maybe tomorrow after I get done with my errands and homework.




Incidentally, the entry in the screenshot that says "anon" and "N.A." is an attached file that isn't associated with any post or PM. They happened accidentally when I was uploading something but didn't finish the post. Apparently the forum doesn't associate them with anything until the post/PM has been submitted, so they were stranded with no data in the DB to link them to a user. In the admin panel they turn up as belonging to "Guest" and can be manually deleted. There are also some buttons to "clean up" the forum, so I imagine one of them can probably deal with such items.
Attachments
package_list.php.gz
The php script
(1.58 KiB) Downloaded 535 times
fancy.png
The fancy view
(64.06 KiB) Downloaded 589 times
basic.png
The basic view
(30.14 KiB) Downloaded 472 times
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

lluamco
Posts: 208
Joined: Fri 16 Mar 2007, 09:10
Location: Banyoles, Spain

#25 Post by lluamco »

Hi Tom.
Thank you for your nice and useful program! I have detected one problem, though. Pets located at dotpups.de do not appear. It seems that the source dotpups.de is not recognized by the search engine. Or am I misunderstanding something?
Thanks

User avatar
tombh
Posts: 422
Joined: Fri 12 Jan 2007, 12:27
Location: Bristol, UK
Contact:

#26 Post by tombh »

@HairyWill: What a neat idea! I'll look into it :)

@Pizzasgood: Umm, I can't see any reason for there being a second path entry for each item, that's just the way MU's original script is. At the moment though the search script takes the SOURCE field from the first path entry and the DIRECTORY field from the actual URL of the text file that all the entries are contained within (e.g. http://127.0.0.1/puppytest/package_list.php?view=basic). Therefore, I don't think the second path entry is actual used at all!

Well it looks great :)

Maybe one thing to consider could be saving the output as a stand alone file, if the file was accessed a lot then it would incur less CPU load if PHP/SQL calls weren't being run all the time -- obviously then you'd need a cron job to update that text file every so often. But hopefully we can set it up so that Petch never calls your script directly, but calls a copy of it stored on puppylinux.org.

@lluamco: Thanks :) Can you give some examples?
[url=http://www.tombh.co.uk/colour_memories/contribute.php]Colour-Memories Database[/url]

lluamco
Posts: 208
Joined: Fri 16 Mar 2007, 09:10
Location: Banyoles, Spain

#27 Post by lluamco »

Hi again, Tom.
My observation started when using petch hplip
The output is

SOURCE | DIRECTORY | FILENAME (size)
[ 1] - puppylinux.ca | members | hplip_for_puppy_217-1.7.4a-i486.pet (46729952)
[ 2] - puppylinux.ca | members | hplip_for_puppy-1.7.4a.pet (32093140)
Please type the number of the package you would like to fetch:

only 2 very old packages appear, and none of the new ones located at

http://dotpups.de/puppy4/dotpups/Printi ... for_dingo/
and its mirror
http://puppyfiles.ca/dotpupsde/puppy4/d ... for_dingo/
appear. However, afterwards I realized that MU is working on this particular puppy4/dotpups directory, so it should be good waiting... (thanks Canery!)

But then I tried also to petch cups, and I obtained this output:

SOURCE | DIRECTORY | FILENAME (size)
[ 1] - puppylinux.ca | puppyfiles | cups-1.1.23-1.pet (1606782)
[ 2] - puppylinux.ca | puppyfiles | cups-1.2.11.pet (1557475)
[ 3] - puppylinux.ca | puppyfiles | cups_espgs_gutenprint-1.1.23-8.15-5.0.0-2.pet (5118201)
[ 4] - puppylinux.ca | puppyfiles | cups_pdf-0.8.pet (8105)
[ 5] - puppylinux.ca | puppyfiles | cups_espgs_gutenprint-1.1.23-8.15-5.0.0-2.pet (1543728)
[ 6] - puppylinux.ca | puppyfiles | cups_espgs_gutenprint-1.1.23-8.15-5.0.0.pet (5118557)
[ 7] - puppylinux.ca | puppyfiles | cups_espgs_gutenprint_DEV-1.1.23-8.15-5.0.0.pet (1112408)
[ 8] - puppylinux.ca | puppyfiles | cups_pdf-0.2.pet (2215)
[ 9] - puppylinux.ca | puppyfiles | cups_pdf-0.3.pet (2388)
[10] - puppylinux.ca | puppyfiles | cups-1.1.23.pet (1175167)
[11] - puppylinux.ca | puppyfiles | cups-1.3.3.pet (1701989)
[12] - puppylinux.ca | puppyfiles | cups_pdf-0.8.pet (8173)
[13] - puppylinux.ca | puppyfiles | epdfview-0.1.6-4cups1123.pet (66464)
[14] - puppylinux.ca | puppyfiles | ghostscript-8.15.2-4cups1123.pet (1916649)
[15] - puppylinux.ca | puppyfiles | gtk+-2.12.1-4cups1123.pet (1498926)
[16] - puppylinux.ca | puppyfiles | gutenprint-5.0.1-4cups1123.pet (1658231)
[17] - puppylinux.ca | puppyfiles | libgnomecups-0.2.2-4cups1123.pet (19549)
[18] - puppylinux.ca | puppyfiles | libgnomecups-0.2.2.pet (19581)
[19] - puppylinux.ca | puppyfiles | libgnomeprint-2.18.2-4cups1123.pet (220590)
[20] - ibiblio | pet_packages-2 | cups_espgs_gutenprint-1.1.23-8.15-5.0.0-2.pet (1.5M)
[21] - ibiblio | pet_packages-2 | cups_espgs_gutenprint-1.1.23-8.15-5.0.0.pet (4.9M)
[22] - ibiblio | pet_packages-2 | cups_espgs_gutenprint_DEV-1.1.23-8.15-5.0.0.pet (1.1M)
[23] - ibiblio | pet_packages-2 | cups_pdf-0.2.pet (2.2K)
[24] - ibiblio | pet_packages-2 | cups_pdf-0.3.pet (2.3K)
[25] - ibiblio | pet_packages-4 | cups-1.1.23.pet (1.1M)
[26] - ibiblio | pet_packages-4 | cups-1.3.3.pet (1.6M)
[27] - ibiblio | pet_packages-4 | cups_pdf-0.8.pet (8.0K)
[28] - ibiblio | pet_packages-4 | epdfview-0.1.6-4cups1123.pet (65K)
[29] - ibiblio | pet_packages-4 | ghostscript-8.15.2-4cups1123.pet (1.8M)
[30] - ibiblio | pet_packages-4 | gtk+-2.12.1-4cups1123.pet (1.4M)
[31] - ibiblio | pet_packages-4 | gutenprint-5.0.1-4cups1123.pet (1.6M)
[32] - ibiblio | pet_packages-4 | libgnomecups-0.2.2-4cups1123.pet (19K)
[33] - ibiblio | pet_packages-4 | libgnomecups-0.2.2.pet (19K)
[34] - ibiblio | pet_packages-4 | libgnomeprint-2.18.2-4cups1123.pet (215K)
[35] - ibiblio | pet_packages-3 | cups-1.1.23-1.pet (1.5M)
[36] - ibiblio | pet_packages-3 | cups-1.2.11.pet (1.5M)
[37] - ibiblio | pet_packages-3 | cups_espgs_gutenprint-1.1.23-8.15-5.0.0-2.pet (4.9M)
[38] - ibiblio | pet_packages-3 | cups_pdf-0.8.pet (7.9K)
[39] - puppylinux.ca | tpp | cups_addon.pet (13406385)
[40] - puppylinux.ca | tpp | libpoppler-libcups-patch-1.00.pet (744693)

where NONE of the sources is dotpups.de. That is very strange, since there are cups packages in MU's repository.

Still another very minor detail. If you invoke petch without arguments, its output is
Usage: fetch [package to search for]
fecth should be petch, right?

Thanks again,
Lluis

Caneri
Posts: 1513
Joined: Tue 04 Sep 2007, 13:23
Location: Canada

#28 Post by Caneri »

Hi Lluis,

puppyfiles.ca here in Canada is the mirror for MU's dotpups.de in Germany and should be exactly the same as MU's main server.

When you see this
[ 1] - puppylinux.ca | puppyfiles | cups-1.1.23-1.pet (1606782)
this file is exactly the same on dotpups.de.

I'll let MU speak on that to be sure.

Petch is evolving so some directories/package lists will be updated in the near future...but tombh,MU will be the ones to respond to this.

Best,
Eric
[color=darkred][i]Be not afraid to grow slowly, only be afraid of standing still.[/i]
Chinese Proverb[/color]

lluamco
Posts: 208
Joined: Fri 16 Mar 2007, 09:10
Location: Banyoles, Spain

#29 Post by lluamco »

Thanks Eric,
I was erroneously expecting to read dotpups.de as Source, my mistake.
Thanks again for clarifying and sorry for the noise...

Caneri
Posts: 1513
Joined: Tue 04 Sep 2007, 13:23
Location: Canada

#30 Post by Caneri »

Hi lluamco,

Questions are not noise but helpful.

Now I see that trying to pull a specific pet form puppyfiles.ca/dotpups.de is not working for me also.

My above post is wrong..the phrase [ 1] - puppylinux.ca | puppyfiles | cups-1.1.23-1.pet (1606782) is coming from puppylinux.ca/puppyfiles..sorry...hey I'll figure this out yet..;-)

Best,
Eric

PS I'll keep my mouth shut and listen for awhile..then I may be of use.
[color=darkred][i]Be not afraid to grow slowly, only be afraid of standing still.[/i]
Chinese Proverb[/color]

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#31 Post by Pizzasgood »

Maybe one thing to consider could be saving the output as a stand alone file
Yes, I was thinking about doing that too. I probably would have already, but I had planed to do the bulk of the work on Sunday afternoon. After fiddling with my network settings around noon I had to wait a while before being able to access the campus network, so I thought I'd play a quick game of Nethack. Wound up being anything but quick. Five hours or so later I finally died, after accumulating a score six times higher than my previous high score. What ultimately led to my demise was accidentally reading a scroll of Destroy Armor, which ruined my +4 robe, thus slashing my defenses and killing my ability to use magic. :( Then the day was nearly over.

I'll clean stuff up and set it to output a file tonight, add some formatting to the fancy view, and add a third mode to output the data from the fancy view in a more usable form in case we ever want to set up something like that attachment->repo thing we talked about earlier. Then I'll upload it all to my website so you can play with it and make sure it works.
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#32 Post by Pizzasgood »

Test forum with info typed when I wasn't half sleeping over here:
http://www.browserloadofcoolness.com/pu ... ic.php?t=3

Needs testing and examination before I email it to John.

Notice that the ?view parameter has been renamed ?mode.

This version is also CLI compatible. It can take the mode as a normal parameter, so that it can be used with Cron properly.

Added an option to output to a file. Disabled by default when viewed through the web, but enabled (and always enabled) when run from CLI.

Go ahead and upload as much as you want over there to test it. I'm gonna delete it all when we get done so spam away.

Let me know if there's anything you think needs tweaking. Preferably over here.

Now I go to enjoy sleep time. 'night.
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
tombh
Posts: 422
Joined: Fri 12 Jan 2007, 12:27
Location: Bristol, UK
Contact:

#33 Post by tombh »

@lluamco:
When you see this
[ 1] - puppylinux.ca | puppyfiles | cups-1.1.23-1.pet (1606782)
this file is exactly the same on dotpups.de.
That's true. And yes, MU is still in the process of making package lists for all his directories.

@pizzasgood: Well, it all looks fantastic. I uploaded a new attachment (petch-0.3.pet as it happens), copied the package list over to puppylinux.org and now

Code: Select all

petch petch
returns
# petch petch
searching ...

SOURCE | DIRECTORY | FILENAME (size)
[ 1] - browserloadofcoolness.com | | petch_03.pet (1811)
[ 2] - puppylinux.ca | tpp | petch-0.3.pet (1811)
Please type the number of the package you would like to fetch:
Obviously I'll tidy up the formatting -- browserloadofcoolness,com breaks the nice neat columns! Also maybe you could put something useful in that second path entry bit, maybe the thread title or the contributors forum name? That info would fit into the DIRECTORY field.

So what about the HTML fancy view? I imagine the list is going to be gigantomungous (ie long) on the murga forum. Perhaps too long to show on a single page -- i,e. endless page-loading? Maybe the HTML view should require an extra search query argument to filter things down a bit? I guess the XML view is fine as it's not for human eyes.

I wouldn't be surprised though that some unforeseen shenanigans arise once plugged in here at the murga forum, maybe keep it low key for a couple of days testing?

On an aside, one day there will only be petget right? It'll contain petch's code and other goodies that have accumulated over the years in the forum. And we'll take repository organisation and searching into account when re-coding dir2pet, therefore asking who put the pet together, what version of Puppy is it for, and so on. Basically I mean, one day, we're all gonna get together and come up with a complete package management solution, aren't we? Yes :)
[url=http://www.tombh.co.uk/colour_memories/contribute.php]Colour-Memories Database[/url]

User avatar
Pizzasgood
Posts: 6183
Joined: Wed 04 May 2005, 20:28
Location: Knoxville, TN, USA

#34 Post by Pizzasgood »

Doh, I should have though about that. Yes, there are several thousand packages on this forum. That would be a long nasty list. In that case, I think I had better just split this thing in two. Have one to create the basic view and XML, and a separate one to do the fancy. Then it can have pagination and searching without worrying about affecting the stability and complexity of the other two modes, which should almost never need modification.

I'll split it out on Tuesday probably, then email the basic/xml version to John and see if he'll add it. I'll probably advise him to set it up so people can't trigger an update over the web to avoid the potential for attacks. I'll work on the fancy view when I get time. I have a largish project for my "Computer Arch and OS" class that I need to start working on this week.

(I'm visiting my grandma and grandpa this weekend, so no working this weekend other than my combinatorics homework, which I finished earlier today).



I'll also change that second field to the user name before I split them.




Yes, I think long-term it would be good to set up a more descriptive repository system and have petch-like functionality built into PetGet.
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]

User avatar
Béèm
Posts: 11763
Joined: Wed 22 Nov 2006, 00:47
Location: Brussels IBM Thinkpad R40, 256MB, 20GB, WiFi ipw2100. Frugal Lin'N'Win

#35 Post by Béèm »

Just discovered this handy little program.
I tried to trick it by issuing.

Code: Select all

petch *
but I got nicely

Code: Select all

[95] - puppylinux.ca | puppyfiles     | gxmessage-2.6.2.pet (15660)
[96] - puppylinux.ca | puppyfiles     | hardinfo-0.4.2.2.pet (256334)
[97] - puppylinux.ca | puppyfiles     | hexedit-1.2.10.pet (62522)
[98] - puppylinux.ca | puppyfiles     | hv3-20070702.pet (579579)
[99] - puppylinux.ca | puppyfiles     | ical-2.3.1patched.pet (161002)
100 results reached, no more results will be shown.
Please type the number of the package you would like to fetch: ^C
# 
Somebody made the remark that it would be nice to be able to indicate a save to directory, but there is no need for this.
Just open rxvt and go to the folder where you want the package to be downloaded and retrieve it.

Also a remark was made that an uninstall should be nice, but there is no need for it either as petch calls the regular puppy package manager and you can uninstall with it.

Just one thing, when a package is retrieved again, there is no checking if exists already. It's simply overwritten.
Not a disaster for small packages but a waist of time and bandwidth for large downloads. Specially when one is on a dial-up connection.
So here is room for improvement.
Time savers:
Find packages in a snap and install using Puppy Package Manager (Menu).
[url=http://puppylinux.org/wikka/HomePage]Consult Wikka[/url]
Use peppyy's [url=http://wellminded.com/puppy/pupsearch.html]puppysearch[/url]

Post Reply