How To: Search Unbuntu repository in Bash?

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Post Reply
Message
Author
User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

How To: Search Unbuntu repository in Bash?

#1 Post by sunburnt »

I looked at wget and didn`t see any find or search type features.
It`s very possible I missed understanding some of the options.

I tried this to see if I could get a dir. listing I could parse:

Code: Select all

wget -r -A deb --spider http://ubuntu.mirrors.pair.com/archive/pool/universe/s/*
I need to find and verify a files existence ( like a library file ).
And to get a dir. listing to do "grep" on is good for unknown file names.

Search an Ubuntu mirror like:

Code: Select all

http://ubuntu.mirrors.pair.com/archive/pool/universe/
technosaurus gave me a web page file that you can manually do a search.
But I need my app. to be able to automatically find and download files.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#2 Post by amigo »

http is not really searchable. You have to use wget to download an html page of the site and then parse entries out of that -a very tedious thing, indeed.
If you already know the name of the file, then you can simply use the exit status of the wget command to tell whether the download was successful. But if you need to dig through possible/unknown subdirectories, then you need to do as above suggested. ftp is also not really searchable, but there a couple of ways to get an 'ls' type listing of the site. Some sites accessed using http can also be accessed using ftp, but mileage varies greatly.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#3 Post by jpeps »

You can grab an index, and search it offline:

ls-lR.gz

http://archive.ubuntu.com/ubuntu/

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#4 Post by sunburnt »

Hey jpeps; I looked at that list and wow, what a mess! I have several other lists, one just files and one with a full description.
I can`t find where I got them, they don`t make it easy. Ultimately I need the download url for wget, which the full one gives.

Hi amigo; Parsing a download mirror`s index page for a listing isn`t so bad really. But the main site`s pages are a maze.
Ubuntu has libs in specific dirs., the apps. have specific libs in with them. This makes it a little troublesome to find the libs.


It would be really nice if sites had text files of their pages, but it`s obvious that Ubuntu wants only apt-get to auto. download.


### Qs: Is Slackware`s download site straight forward and sanely organized? And does it have all the files that Ubuntu has?
.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#5 Post by jpeps »

sunburnt wrote:Hey jpeps; I looked at that list and wow, what a mess! I have several other lists, one just files and one with a full description.
everything is there; parse it.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#6 Post by sunburnt »

jpeps; Apparently the Murga Puppy site won`t take attachments now...

I tried several times to post the file I spoke of but no luck.
Looking at the upload limits, it says 0 MB allowed! :lol:
My gzipped file is 6.8 MB, so it may be too big, I don`t know.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#7 Post by jpeps »

It's the entire database, so it's huge...unzipped 126M.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#8 Post by sunburnt »

Here`s the Geany package`s info.:

Code: Select all

Package: geany
Priority: optional
Section: universe/devel
Installed-Size: 6492
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Damián Viano <des@debian.org>
Architecture: i386
Version: 0.18-1
Depends: libatk1.0-0 (>= 1.20.0), libc6 (>= 2.4), libcairo2 (>= 1.2.4), libfontconfig1 (>= 2.4.0), libfreetype6 (>= 2.2.1), libgcc1 (>= 1:4.1.1), libglib2.0-0 (>= 2.18.0), libgtk2.0-0 (>= 2.17.7), libpango1.0-0 (>= 1.14.0), libstdc++6 (>= 4.1.1)
Suggests: libvte9, doc-base
Filename: pool/universe/g/geany/geany_0.18-1_i386.deb
Size: 2366426
MD5sum: 7bbd2f5f369e7cb4c1b335e2ef2139ce
SHA1: 796906f02afad38ac89043642cc4c9f114b89515
SHA256: 58010219836361d9d9f50afb53d9174c0896a3c0cdcf6efcbbe786e9d3e76c9d
Description: A fast and lightweight IDE
 Geany is a small and lightweight integrated development environment.
 It was developed to provide a small and fast IDE, which has only a
 few dependencies from other packages. It is using only the GTK2 toolkit
 and therefore you need only the GTK2 runtime libraries to run Geany.
 .
 The basic features of Geany are:
  - syntax highlighting
  - code completion
  - auto completion of constructs like if, for and while, XML and HTML
  - call tips
  - folding
  - many supported filetypes like C, Java, PHP, HTML, Python, Perl, Pascal
  - symbol lists
  - embedded terminal emulation
 .
Homepage: http://www.geany.org
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
It has the description which I need for the GUI, and the other data.
As you can see it`s a breeze to parse the data.

Unfortunately I cannot find where I got this file from!!!
.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#9 Post by jpeps »

Yes, but it doesn't tell you what's available. You can use "man geany" for a description of geany. Here's the database search for geany, 2012:

1936302 Mar 19 2012 geany-common_0.21.dfsg-1ubuntu4_all.deb
11739 Mar 19 2012 geany_0.21.dfsg-1ubuntu4.debian.tar.gz
2236 Mar 19 2012 geany_0.21.dfsg-1ubuntu4.dsc
1011790 Mar 19 2012 geany_0.21.dfsg-1ubuntu4_amd64.deb
858418 Mar 19 2012 geany_0.21.dfsg-1ubuntu4_armel.deb
856420 Mar 19 2012 geany_0.21.dfsg-1ubuntu4_armhf.deb
1016568 Mar 19 2012 geany_0.21.dfsg-1ubuntu4_i386.deb
994392 Mar 19 2012 geany_0.21.dfsg-1ubuntu4_powerpc.deb
4176730 Jan 5 2012 geany_0.21.dfsg.orig.tar.gz

If I search for an entry, then I get all the specifics on a download page:

http://pkgs.org/ubuntu-12.04/ubuntu-uni ... l.deb.html
geany-common_0.21.dfsg-1ubuntu4_all.deb Download

geany-common - fast and lightweight IDE -- common files

Geany is a small and lightweight integrated development environment. It was developed to provide a small and fast IDE, which has only a few dependencies from other packages. It is using only the GTK2 toolkit and therefore you need only the GTK2 runtime libraries to run Geany.

The basic features of Geany are:

- syntax highlighting

- code completion

- auto completion of constructs like if, for and while, XML and HTML

- call tips

- folding

- many supported filetypes like C, Java, PHP, HTML, Python, Perl, Pascal

- symbol lists

- embedded terminal emulation

This package contains arch independent files.
...

Distribution: Ubuntu 12.04.1 LTS
Repository: Ubuntu Universe i386
Package name: geany-common
Package version: 0.21.dfsg-1ubuntu4
Binary package: geany-common_0.21.dfsg-1ubuntu4_all.deb
Source package: geany
Download size: 1,85 MB

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#10 Post by sunburnt »

jpeps; Interesting, but this is run from my GUI, there`s no download page.
Or I should say... The text file`s easier to parse than html as amigo mentioned.

The only difference is it shows other architectures. But the only one interesting is amd64.
All you need to do is substitute amd64 ( or any other ) for i386 and viola, a different arch.

Also if you haven`t downloaded the package, then there`s no man page to read, right?

There are about a dozen or two ways to do this that I`ve seen so far. Simple is good.

Here`s the Ubu. backend info. parser script I wrote today:

Code: Select all

#!/bin/sh
##########	File Package Info.			Terry Becker	Oct. 15  2012

####	Usage:  . pkg-info (Pkg. Name)


[ "$1" ]|| exit

#=====================================================	Ubuntu Section.
[ "$2" ]|| Mirror=http://mirrors.us.kernel.org/ubuntu/
FileList=${0%/*}/Ubuntu_Universe-Pkgs.txt
																										# find pkg.
StN=`grep -n -m1 "^Package: $1$" $FileList`
[ "$StN" = '' ]&& echo -e "\n###  ERROR:  Did Not Find Package:  $1\n" && exit
StN=${StN/:*}
Info=`sed $StN,$((StN+100))'!d' $FileList`

EndN=`echo "$Info" |grep -n -m1 '^Bugs:'`
EndN=$((${EndN/:*}-2))
Info=`echo "$Info" |sed 1,$EndN'!d'`
																										# parse info.
PathPkg=`echo "$Info" |grep '^Filename:'`
PathPkg=${PathPkg#*: }
MD5sum=`echo "$Info" |grep '^MD5sum:'`
MD5sum=${MD5sum#*: }
Version=`echo "$Info" |grep '^Version:'`
Version=${Version#*: }
Arch=`echo "$Info" |grep '^Architecture:'`
Arch=${Arch#*: }
Depends=`echo "$Info" |grep '^Depends:'`
Depends=${Depends#*: }
PkgSize=`echo "$Info" |grep '^Size:'`
PkgSize=${PkgSize#*: }
InstSize=`echo "$Info" |grep '^Installed-Size:'`
InstSize=${InstSize#*: }
Desc=`echo "$Info" |sed -n '/Description/,$p'`
#=====================================================

User avatar
RSH
Posts: 2397
Joined: Mon 05 Sep 2011, 14:21
Location: Germany

#11 Post by RSH »

Hi.

I don't know if this will help. The attached .tar.gz contains all directories for http://archive.ubuntu.com and http:://ubuntu.mirrors.pair.com

I have extracted this from the ls-lR.gz file.

RSH
[b][url=http://lazy-puppy.weebly.com]LazY Puppy[/url][/b]
[b][url=http://rshs-dna.weebly.com]RSH's DNA[/url][/b]
[url=http://murga-linux.com/puppy/viewtopic.php?t=91422][b]SARA B.[/b][/url]

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#12 Post by sunburnt »

Hi RSH; I just posted after you in the SFS files thread too!

I`ll look at your attachment, the file I have is smaller than ls-lR by 100 MB!
This makes it much easier to include with my "package build and dnld." GUI.
It has most all the info. I need, but further along I may need to use ls-lR.

Again, ultimately what I need is the download URL for a given app`s exec.
Then my script runs "ldd" to get the execs. missing libs., and here the pkg.
list will give the dnld. URL and my script dnlds. them and checks ldd again.
Using ldd solves the duplication of libs. for any given host Linux O.S.

So for truly more portable AppPkg`s, use a minimal O.S. like Tiny Core.
This way most of the needed libs. would be included in the AppPkg.
I may make an option to include all libs., or allow selection from a list.

Post Reply