A local /usr/share/docs/packages.htm

[sc0ttman]

Suggestion: An even easier way to download packages from online

SUMMARY:

A shell script to generate a static HTML page, maybe during a woof build
which lists a table of packages for each supported repo, with each pkg
name being clickable link, straight to pkg (.deb etc) download.

(This is a much simpler alternative to hosting a cool package search site
online somewhere, and a nice page to link to from the default homepage)

DETAILS:

One of Puppies main goals is ease of use.
Puppy often goes the extra mile to make things, easier, simpler, faster.

Many users (not myself, ever) go to Debian/Ubuntu/Slackware the big
package websites like https://packages.debian.org/ or
https://packages.ubuntu.com/) to download pkgs manually.

For some, this is reliable, and "just works" .. But it's a pain, and those
sites are not at all user friendly for non-Linux users or newbies.

Plus, you have to know about them and find them first..

!! For some, their preferred browser might not even work with those sites !!

Take for example, the Debian 'Ace of Penguins' package page:

https://packages.debian.org/stretch/gam ... f-penguins

It lists some of deps that might break a Puppy users system.

The pkg itself is way down at the bottom, where it lists lots of debs, most for
other architectures, and also lots of irrelevant guff on the right.

Finding the correct deb is not easy for newbies, and a needless hassle for regulars..

...But we know we want to download the .deb for the running $ARCH..
.. and woof worked out which mirrors are working during build time..

Maybe links to the packages can be auto-generated with something like:

Code: Select all

<a href="http://$MIRROR1/$DISTRO_COMPAT_VERSION/$DBIN_ARCH/$PKGNAME/download">i386</a>

So ... stay with me ...

Instead of the user:

- finding the Debian/Ubuntu/Slack/other packages site themselves,
- then finding the right DISTRO_BINARY_COMPAT version (stretch, trusty, wheezy, xenial, tahr, etc),
- then finding the package,
- then finding the right architecture,
- then choosing a mirror,
- and finally getting their download...

We could have a link in /usr/share/docs/index.htm to a page which
was generated in woof at build time, and contains a simple listing
of all pkgs from that puppies repo files.

For example, /usr/share/docs/packages.htm

Which could list ALL packages available in the repos, each repo in a
separate HTML table, with name, deps, size, description, and a clickable
link to download the package (woof knows the mirrors that worked at
build time, at least!).

Any deps listed could simply be anchor links (like #pkgname),
which takes the user straight to the package on that same page! How easy..

As in my other post about the default Puppy homepage, we know that
at build time, woof knows (more or less) enough about the Puppy being
built to generate links to specific, relevant web pages..


.... This (mostly generation of a big long HTML table in woof
somewhere at build time) may be a great way to easily download
pkgs from all over the place , and save Puppy users having to trawl
around the interwebz..

---- Improvements ----

However, lots of long table are boring, and hard to scroll around, so some
'accordion' CSS/JS could be used to make each repo table expand/contract
when its Heading is clicked ..

Same for package entries themselves - click a package row, it could
expand/contract to show/hide more details..

(So, something like the pkgs.org interface, but a local single HTML file,
listing all locally installed repos, and generated by a script)

The script that generated this packages.htm page could be re-ran to keep
it up to date (if its generated from the local config and repo files anyway)..

... Just a suggestion..

[sc0ttman]

Now more than just a suggestion...

DEMO ONLY

Some links work (pets download OK), but some don't..

The case statement which builds URLs need fixing/finishing..

Screenshot:


Larger image

Save the code below as make_packages_page.sh:

Then do:

Code: Select all

chmod +x ./make_packages_page.sh

Code: Select all

#!/bin/ash
#
#	makepackagespage.sh - builds an HTML page listing all packages in 
#						  all available repo on the running system
#

. /etc/DISTRO_SPECS
. /root/.packages/PKGS_MANAGEMENT

rm -f /tmp/packages.htm

# create our new page
echo '<!DOCTYPE html>
<html>
<head>
<title>All Software Packages</title>
<style>
body {
font-size: 15px;
padding: 0;
margin: 0;
}
.intro {
background-color:#FFF3A3;
font-weight: bold;
margin: 0 auto;
margin-left:28px;
margin-right:28px;
padding:10px;
}
.repo {
margin-bottom: 28px;
}
table {
background-color:#EFEFEF;
margin: 0 auto;
margin-top: 20px;
width: 98%;
}
th {
background-color: #2D98FF;
color: white;
font-size:20px;
text-align: center;
padding-top:10px;
padding-bottom:10px;
}
tr {
margin-top:5px;
margin-bottom:5px;
padding-top:5px;
padding-bottom:5px;
}
.n {
font-weight: bold;
padding-left:6px;
padding-right:6px;
}
.s {
padding-right:6px;
}
.c {
padding-right:6px;
}
.d {
width: 50%;
min-width: 50%;
padding-right:6px;
}
.e {
padding-right:6px;
}
</style>
</head>
<body>
<center>
<h1>'$DISTRO_NAME' Packages</h1>
<div class="intro">
Here are all the packages available for your system.
</div>
</center>' > /tmp/packages.htm

pkgurl=''

for RF in $PKG_REPOS_ENABLED
do
	echo '<div class="repo_block">' 	>> /tmp/packages.htm
	
	# create container and table for this repo
	echo '<div class="repo">'			>> /tmp/packages.htm

	echo '<table>'						>> /tmp/packages.htm
	echo '<thead>'						>> /tmp/packages.htm
	echo '<tr>'							>> /tmp/packages.htm
	echo '<th colspan="5">'				>> /tmp/packages.htm
	echo $RF							>> /tmp/packages.htm
	echo '</th>'						>> /tmp/packages.htm
	echo '</tr>'						>> /tmp/packages.htm
	echo '<tr>'							>> /tmp/packages.htm
	echo '</thead>'						>> /tmp/packages.htm

	echo "Building pkg list for `basename $RF`, please wait."
		
	case $RF in
		Packages-puppy*noarch*) pkgurl="http://ftp.nluug.nl/os/Linux/distr/puppylinux/pet_packages-noarch";;
		Packages-puppy*common*) pkgurl="http://ftp.nluug.nl/os/Linux/distr/puppylinux/pet_packages-common";;
		Packages-puppy*slacko*) pkgurl="http://distro.ibiblio.org/puppylinux/pet_packages-slacko$DISTRO_COMPAT_VERSION";;
		Packages-*slackware*) 	pkgurl="http://slackware.mirrors.tds.net/pub/slackware/slackware-$DISTRO_COMPAT_VERSION";;
		Packages-*blah*) 		pkgurl="http://distro.ibiblio.org/more/links";;
		Packages-other*) 		pkgurl='' ;;
	esac
	
	cat /root/.packages/$RF | while read pkgline 
	do
		# get pkg details
		pkgname=`echo $pkgline | cut -f2  -d'|'`
		pkgfile=`echo $pkgline | cut -f8  -d'|'`
		pkgcat=` echo $pkgline | cut -f5  -d'|'`
		pkgdir=` echo $pkgline | cut -f7  -d'|'`; [ "$pkgdir" != '' ] && pkgdir="$pkgdir/"
		pkgsize=`echo $pkgline | cut -f6  -d'|'`
		pkgdeps=`echo $pkgline | cut -f9  -d'|'`
		pkgdesc=`echo $pkgline | cut -f10 -d'|'`
		
		# build our pkg URL
		pkgurlfull="$pkgurl/$pkgdir$pkgfile"
		
		pkgrow=''
		# each each pkg to its repo table
		pkgrow="$pkgrow"'<tr>'
		pkgrow="$pkgrow"'<td class="n">'
		pkgrow="$pkgrow"'<a href="'$pkgurlfull'">'$pkgname'</a>'
		pkgrow="$pkgrow"'</td>'
		pkgrow="$pkgrow"'<td class="c">'
		pkgrow="$pkgrow"''$pkgcat
		pkgrow="$pkgrow"'</td>'
		pkgrow="$pkgrow"'<td class="s">'
		pkgrow="$pkgrow"''$pkgsize
		pkgrow="$pkgrow"'</td>'
		pkgrow="$pkgrow"'<td class="d">'
		pkgrow="$pkgrow"''$pkgdesc
		pkgrow="$pkgrow"'</td>'
		pkgrow="$pkgrow"'<td class="e">'
		pkgrow="$pkgrow"`echo $pkgdeps | head -c 50`
		pkgrow="$pkgrow"'</td>'
		pkgrow="$pkgrow"'</tr>'
		
		#echo "$pkgname $pkgurlfull $pkgcat $pkgsize $pkgdeps $pkgdesc" | head -c 78
		
		echo "$pkgrow" >> /tmp/packages.htm
	done

	# finished adding pkgs, close table and container
	echo '</table>'	>> /tmp/packages.htm
	echo '</div>'	>> /tmp/packages.htm
done

# finish our page
echo '</body></html>' >> /tmp/packages.htm

# final checks
if [ -f /tmp/packages.htm ];then
	mv /tmp/packages.htm /usr/share/doc/packages.htm
	echo "Page created: /tmp/packages.htm"
	# open it 
	defaultbrowser /usr/share/doc/packages.htm
else
	echo "Page NOT created!"
	exit 1
fi


exit 0

EDIT: updated code above, less spacing in final HTML file.

[peebee]

Interesting idea....

Well it worked and created a nice webpage (on LxPupSc).....

but oh-boy took a longggggg time and created a 3.8MB file so not something you'd want to add to a build...

The repos included should probably be constrained to be the same as PPM by:
$PKG_REPOS_ENABLED
from /root/.packages/PKGS_MANAGEMENT ??

Cheers
peebee

[Keef]

Ran this on Quirky, and it also took a while to generate. Slow to load too. 32MB!
The URLs are not being made properly though.
eg First link is for 915resolution, but the URL is:

Code: Select all

file:///915resolution-20110617-x86_64.pet

All other links start with file:/// as well.

[sc0ttman]

Ran this on Quirky, and it also took a while to generate. Slow to load too. 32MB!
The URLs are not being made properly though.
eg First link is for 915resolution, but the URL is:

Code: Select all

file:///915resolution-20110617-x86_64.pet

All other links start with file:/// as well.

Yeah its just demo code...

The case statement needs to be fixed to account for the various repos,
and their various URL syntaxes, vars, etc...


On Slacko 6.9 with the code above, I get a 2133k file, took about 4 mins!

Possible improvements

- build different HTML pages for each repo..
- use *something* like the following instead of reading each repo line:

Code: Select all

cut -f2,5,6,7,8,9,10 -d'|' ~/.packages/Packages-puppy-slacko14.2-official | sed -e "s/^/<td>/" -e "s/|/<\/td><td>/g" -e "s/\$/<\/td>/g"

The above will cut the fields we need, then convert ^ (start of line), the | and also
$ (end of line) to HTLM <td> stuff .. It is much faster as it does the file all in one
go (not line by line).. but won't give you links/urls ..

EDIT:

FWIW, I cleaned up the file a little, keeping the HTML as concise as
possible, got the file down to ~1.8mb, then made an sfs of the .htm
file to see (very roughly) how small it would compress:

# du -h packages.sfs
224K packages.sfs

EDIT:

Updated to use Peebees $PKG_REPOS_ENABLED suggestion.

Could also remove deps and/or description to make things faster and smaller.

[sc0ttman]

Updated, in case anyone is interested in doing something with this..

- separate repos into their own pages
- generate an index page to choose repos

The repos index page (/usr/share/doc/packages.htm):


.. still slow.. still large HTML files (but they compress well, and could make a
small PET package).

Attached a fake gz ...

Download and remove the extension, then

Code: Select all

chmod +x makepackagepages.sh
mv ./makepackagepages.sh /usr/sbin/make_packages_pages.sh
make_packages_pages.sh

Your files end up in /usr/share/doc/packages*.htm

[sc0ttman]

If I was gonna extend this, I would progressively enhance with JS:

1. Add a search filter
- add a <script> tag just before the closing </body> tag of each page
- use JS to put a search box as the top of each repo page
- inside the <script> tag:
- use JS to get search term from search input field
- hide (add css display:none) to all non-matching <tr> elems
- clear search field to unhide all elems

2. Don't list deps in the main tables
- list deps as data-deps="" in the <tr>s instead
- click a package, get a popup modal with all pkg details
- modal lists deps as links, to easily download package and its deps all in one place
- popup modal can be achieved with a totally centered <div>, plus CSS to hide/show it
- click the DOWNLOAD button in the modal popup to download the pkg

3. Auto generate links to more packages
- as in this thread, generate links to Slackware/Ubu/Deb package sites
- show links to extra packages under the repos list, in same style

With 'progressive enhancement', features 1 and 2 would only appear if the
users browser supported JS, otherwise they would get just the static
HTML pages as they are now.


...Alternatively, the sh script could be altered to generate a PERL .cgi file,
and the page loaded up on http://localhost, and then more dynamic
behaviours and backend cleverness could be added that way (query strings, calling cmds on the system, etc).