Puppy Website: Package Lists

[prit1]

Hello everyone,

I am trying to collect all the package list files available so that this can be integrated as a part of the proposed website under "software search".

Currently I only know the below:
http://puppyfiles.org/dotpupsde/dotpups ... otpups.txt
http://puppyfiles.org/dotpupsde/dotpups ... otpups.txt
http://puppyfiles.org/dotpupsde/dotpups ... otpups.txt
http://puppyfiles.org/dotpupsde/dotpups/dotpups.txt
http://users.ecs.soton.ac.uk/wmd04r/puppy/dotpups.txt

These files normally follow the below layout:
<package location><directory/category><size>

If you know of any package list files that are not mentioned above, can you please provide the below details?
1 - The link to the package list file
2 - Who maintains it
3 - Frequency of updating this package list file.

Thanks,
Prit

[Dingo]

see also my puppy packages wiki:
http://puppylover.netsons.org/dokupuppy/

[tombh]

Also I was wondering how these lists are made, how often and if they do follow a standard syntax, where is this formally defined? I guess these are all probably questions for MU as he designed and wrote PSI.

Prit, I know you mentioned that, whilst making the software search on the new website, you've discovered that the same packages can be found on various lists and repositories, meaning that duplicate results (but from different sources) turn up on some searches. It reminds a little bit of Slackware's repository where you can choose various mirrors to download a package from.

So wouldn't it be great if Puppy formally had something akin to mirrors! I'm not sure how possible that is, maybe we have a couple of servers capable of synchronously reflecting identical repositories but there are too many other smaller repositories with just bits and bobs in (like the forum) at the moment. I think the package lists do provide a very usable and standardised method for finding software. Perhaps it could be easier though for a package to make its way onto one of these lists? Seeing as we now have PSI and Prit's software search on the new website it would be great if most people knew how to quickly and easily get there new dopet to appear in the results of these searches....

[HairyWill]

it would be great if most people knew how to quickly and easily get there new dopet to appear in the results of these searches....

Currently any man and his dog can publish their software using PSI. All you need is to have a wiki account and somewhere on the web to be able to host the pets/pups and a dotpups.txt file. THIS IS SCARY.

This mechanism allows the distribution of untested, unsigned binary software that really could eat your cat and run off with your girlfriend/boyfriend/hampster.

Barry avoids this problem by controlling which packages are made available with petget. I hope that he compiles this software from source himself after downloading it from a reputable repository. I've never met Barry Kauler and I have no good reason to trust him. But you have to start somewhere and he compiled all the other software on my system anyway, for those reasons:
"I trust that the binary software provided provided by Barry Kauler does not contain malicious elements."
The software available through PSI has mainly been placed there by MU. I don't know how much vetting he applies to the software he adds. I probably trust MU and have used binary software provided by many other people on the forum, including through PSI. I have a reasonable understanding of the risks involved and can make an informed choice. Hopefully the number of people using puppy that are NOT able to make an informed choice is increasing.

There are two possible solutions to this problem that I can see;
1)Lock down the list of PSI repositories and make someone we all trust responsible for the list. Force the administrator of each repository to warrant that the software they distribute is safe. This system could be hierarchical to allow one repository to mirror stuff stored on another.
2)Build some trust mechanism relying on digitally signed packages and a hierarchy/ring of trust of digital IDs

This is not an easy problem and I appreciate it raises the barrier for user contribution of software.

On a lighter note here is some data I might like about a package before installing it or to search over:
1)compilation date
2)contributor name
3)target puppy version
4)target processor
5)which puppy trunk versions has it been tested on (this might need to be updateable or force the package to be republished for a new puppy version)
6)short 100ish word description
7)possible link to some small (fixed size) screenshots
8)categories
9)possibly allow user generated tags
10)web link to software home and also to puppy specific maintenance page
11)maybe some classification system for why I should trust it
12)which mirrors it can be found on
13)md5sum of package
14)GUID with each author having a specific namespace, these could be generated by a central system, or a distributed one if you include serverID in the namespace.

Things it might be useful to know about a mirror include
1)the packages it holds
2)its available transfer quota (could be used to avoid going over quota or could be abused to affect a DOS attack)
3)the current download speed to here

As I said to prit the other night

I was talking to MU the other day about the scalability problem of PSI when adding extra mirrors. One of the possible solutions raised was doing some serverside processing to combine the various dotpups.txt files. It seems silly for every client to download the same files and parse them for searching when a server could do it once a day and then provide a REST webservice interface for searching.
...
To sort a list of 10,000 items takes much more than 10 times the time taken to sort 1,000 items.

Proper xml REST can be expensive to unpack on the client but being able to submit search queries and get back short plaintext lists could offer a significant speed boost especially to low powered clients.

If searching was done in this sort of way it would fairly easy to include some mechanism for specifying the desired level of trust required of any candidate mirror.


As you can see I have been thinking about this

[prit1]

I really agree with Tom and Will. We need to come up with a really stable and reliable repository/repositories. And this must be maintained by someone who can be trusted.

To generate all the fields mentioned by Will maybe lot of work. Also, to render it nicely on HTML will be some work. .

Maybe a team can be formed that is responsible for creating and maintaining the repository. They should ensure that the packages are safe, reliable and that the repository is upto date.

Anyway, this discussion might become a very useful one for Puppy.

[SirDuncan]

I also think that we need a better repository/package manager. PSI is wonderful, but could use some refinement. PetGet is absolutely atrocious when compared to the package managers of other distros.

I like how PSI sorts by category, but it currently (unless there is a newer version than what I have) lists sub categories as a separate category (cat1, cat1/sub1, cat1/sub2, etc.). That needs changed to, perhaps, a tree structure of some type. If it were changed to a tree structure, that might also allow us to compress it to a single window. It has a place for a package description, but I don't think any of the packages have anything other than their location and size.

As for the ease of hosting a repository, it is just as easy to host one for Gslapt, is it not? As long as we only put trusted hosts in the default repository list, I think the user should be able to add whatever repository he or she wants. That is, I believe, how it works now.

That's my 2 cents, and 2 cents is worth a lot less than in the past, so take it for what you will.

[raffy]

Can this be implemented using a directory script like one of these? (or perhaps there is a Drupal module?) This way, members can add the package themselves.

[WhoDo]

As for the ease of hosting a repository, it is just as easy to host one for Gslapt, is it not? As long as we only put trusted hosts in the default repository list, I think the user should be able to add whatever repository he or she wants. That is, I believe, how it works now.

Ok, the suggestion was to use servage.net to host packages after we have moved the web site to Hostgator. In that case, if Gslapt does what we want, why not just add that location to Gslapt as our main repository? We can add caneri's server too, for full redundancy. That should also make prit1's software search a lot easier to implement, shouldn't it?

Practically speaking, isn't MU's dotpups repository already on servage.net under puppylinux.org? In that case it would only be necessary to repoint the puppyweb.org domain to that repository to make it available for Gslapt, once the original domain is repointed to Hostgator. Have I got this right or am I working with the lights out?

[MU]

my repository is an own account.
The main domain is http://puppyfiles.org .
Another domain pointing to a subfolder to avoid broken links when I moved to servage, is http://dotpups.de .

If I can help you to set up stuff to make it easier accessable, let me know.
We (BSB-Software) wait for a new root-server, and plan to move everything then.
I also plan to rewrite PSI in GtkBasic, with dialogs to choose from different servers.
The current "mirror-list" is not userfriendly, as it was just an addon-hack.

The base idea of PSI was to let you find all dotpups cluttered around, but not load-balancing or such.
In the beginning it also listed dotpups from grafpup.com and KLHRevolutionist, but those servers are not available any longer.

If someone creates a dotpups.txt for his server it could be added to the Wiki, and so becomes available in PSI.

A more sophisticated approach would allow to choose among "secure, tested" servers, and "unknown" ones.
This certainly will be an issue for us in the second half of 2008.

At moment I am just too busy, I planned to release Muppy 008.3 this weekend, but it will take another week to finish all optimizations I work on.
E.g. I reduced pup_301.sfs (msy_083.sfs) from 168 MB to 102 MB, because Muppy was too slow on machines with 256 MB Ram.
With this change now, it reminds more to the really fast version 007

This took some time, so I could not finish yet to add Gtk2 to Muppy-Server, and so on.
So I am glad, if other people already have a look at the repository and PSI issues

Mark

[SirDuncan]

Ok, the suggestion was to use servage.net to host packages after we have moved the web site to Hostgator. In that case, if Gslapt does what we want, why not just add that location to Gslapt as our main repository? We can add caneri's server too, for full redundancy. That should also make prit1's software search a lot easier to implement, shouldn't it?

Glsapt would probably need altered to handle DotPets and DotPups. It also doesn't organize the programs by category. That wasn't really what I was trying to suggest, though.

What I was getting at is that a Gslapt repository is just as easy to set up as a PSI repository, and Slackware users haven't had a problem with it that I am aware of. However, after reading MU's post and re-reading some of the earlier posts, I realize that the issue is that the default list is being retrieved from the wiki. Since anyone can edit the wiki, the list could be altered to include a malicious repository. The fix for this is easy, just change the place PSI fetches the list from to a non-public edited site whose admin we trust (puppyfiles?).

[wiak]

Currently any man and his dog can publish their software using PSI. . . .
This mechanism allows the distribution of untested, unsigned binary software that really could eat your cat and run off with your girlfriend/boyfriend/hampster.

I really like using Puppy Linux. Indeed I pretty much always use either Puppy or I log into my Ubuntu server.
The thing I really like about puppy, apart from its size and ease of reinstallation, is how simple it is to develop new software on it. However . . . it has also always been on my mind that using Puppy involves an element of trust and risk.

I've been happy to take that risk, since the convenience has outweighed it, but all these recent posts concerning website viruses has made me more than a little wary recently. Partly because of that, I find myself moving more to the Ubuntu family than previously. Bloated though Ubuntu is, it does seem to have rigorous testing procedures, at least for any software in its official repositories.

I guess that is why the Slackware compatible version of Puppy is interesting to me, but of course Puppy itself then becomes fatter, because slackware binaries aren't specially slimmed down. I can't imagine Puppy has the facilities to rigorously vet its own plethora of contributed software though. Personally, I'm more interested in the less-Slackware but slimmer Dingo... but yes... the risks involved...

It is scary and also potentially divisive: whom should you trust? Well-known and respected contributors, to some extent, I suppose (though there is no absolute safety even in that), but what about new developers? - Does Puppy's organisation truly have time and resources for new developer mentoring and vetting. Seems unlikely in practice. It is a big enough job just trying to fix up the web presence.

If only a few developers are allowed to contribute or vet software for Puppy, things are likely to become very limited: there are thousands of packages out there and most would have to be ignored. Perhaps the best that could be done would be to restrict package contributions to Puppy users with at least a few months of forum contributions, and a sizable number of posts.

I also see no reason not to trust Barry Kauler (but do we trust Bill Gates?); beyond that, however, it is anyone's guess really. But, yes, longer term contributors imbue some measure of confidence.

Safer at least, but certainly still not safe. Most people are far too busy to vet other peoples contributions; such work takes a huge amount of time, even for just a few packages.

Or move away from Puppy community-compiled software altogether and use only the likes of official Slackware productions. That would take away a lot of the Puppy pleasure though and make it a considerably less attractive in general.

[wiak]

Of course I've only made around ten posts directly in this particular guise ... Not that that actually means anything.

Though Ubuntu was the only one left standing in a recent Mac versus Vista versus Ubuntu hackers competition, the vista hacker admitted that his exploit could also be adjusted to work on OS X or Linux:

http://www.theregister.co.uk/2008/03/29 ... _standing/
"Nobody can do anything about it, because you're always going to be installing something" that will bypass security, Macaulay, who wore torn blue jeans and a Puma jogging jacket, said with a shrug. "If it's not Java, it'll be something else."

In practice, I much prefer compiling packages for my Puppy Linux installation from source files myself rather than using anyone else's pets or pups, but that is not a practical solution for many people. A medium safety ground has to be found, but Puppy will always be vulnerable, I think, alas.

[HairyWill]

I see two trust issues

1) Authentication - Was this pet really packaged by the person that external metadata says it was? This can be solved by package creators digitally signing the md5sums. This would also require some secure method of importing the public keys of the packagers.

2) Authorisation - Should I trust software provided by the person above. This is the thorny and potentially divisive problem described by wiak.

I agree that we do not have the resources to carry out proper QA and cross checks. Whilst I might be happy trusting the software provided by a number of people in the community for my personal use. It would be reckless and irresponsible for me to warrant to other people about the trustworthiness of this software.

Consider this.
"Who on this forum would you happily give your credit card details to?"
I use the web to do online shopping. I am using the browser installed by whoever assembled my latest favourite pup. I suppose I am trusting them not to have added something malicious to the browser to collect these details.

If microsoft was found to have done this to internet explorer I would probably be able to seek legal address against them. If the mozilla foundation tried to do this people reviewing their source would notice. I don't quite understand how they ensure that binaries are compiled from the stated source. I presume lots of people compile the same source in the same environment and confirm that they get the same binary. If Dougal slipped something unpleasant into 2.14R (for example) how would anyone know except by observing its external behaviour.

I don't think we can completely solve this issue but I do think it would be sensible to be honest and upfront about it.

"This software is provided by Will Davies. You can be assured that it is mine because PSI (written by Mark Ulrich who you might trust) has verified it against my pgp key hosted on the website owned by Barry Kauler (who you might trust) (as long as you believe that your dns access and other internet services have not been tampered with.

I warrant that I have compiled this software from the unaltered publicly available source at mywebsite or the puppy repository."

Reading this vaguely unnerving. Though in reality it is probably very similar to many disclaimers that I click through without reading the fine print.

I guess this adds
15) digitally signed md5sum to my list above.

[tombh]

The trust issue is honestly not one that I've really ever considered, but now that I think about it I can see that it could potentially be a doorway to some very undesirable malicious behaviour. There are of course lots of fangled and clever ways of ensuring the security of software but I think, as HairyWill says, the best response is actually transparency, therefore being as honest and upfront as we can.

Using HairyWill's 15 points, ideally we'd have a web interface that one must log into, fill out some details, upload the file and have the server generate a load of the fields automatically -- therefore md5sums, dates, etc. This is certainly within my, Prit's and Drupal's abilities. But I'm just not sure how successful this would be -- Who would convert all the existing packages over into the new system? How many people would actually contribute their packages like this, choosing instead the simplicity of the forum?

To my mind the emphasis on transparency is the main issue here. Would it not be better to carry on as we are, with the precedent set by PSI, and simply put our efforts into making it as clear as possible that no guarantee can be made for the security of formally archived software? I certainly think it is a bad idea that anyone (and their dog!) can currently get their package appended to the package lists searched by PSI -- it'd be easy to prevent this and if it is not done now it will certainly be done when the WIKI is migrated to Drupal. Then, because of the admins of the servers from which package lists are derived, will there not be a certain consistent measure of vetting?

It's not that we can rely on such vetting for absolute security, but more that it's an already existing line of security that we should make the most of. I think the main thing is that both software searched for by PSI and the website should carry a very clear disclaimer highlighting the sober reality of the situation. This way we are tying into organically established practices and, both acknowledging and encouraging, the open, community spirited will that Puppy inspires.

[HairyWill]

I'm glad that this is making people think and certainly don't want to build barriers that stop this process from happening.

Regarding distributing software via the forum vs PSI.
When I download a pet that is attached to a forum post I am trusting the poster and also whoever has the rights to modify that post presumably John Murga and Flash. I won't get into thinking about any sysadmins that might have access to the server.

When I download a pet from PSI I am trusting, the (anonymous) creator of the pet, the unnamed owner of that mirror (I could probably find out who they are) and also whoever has write access to the list of mirrors (though I can check which mirror I am downloading from). If something does go horribly wrong who do I complain to, who's reputation is tarnished. I blame the owner of the mirror, they direct the blame to wherever they obtained the package from, this could bounce back and forth eroding everyones trust. This scenario would certainly deter me from hosting other peoples binaries. Digitally signed packages could avoid this as there is no way that the mirror can be accused of interfering with the package.

I can't really see how the cost of filling out the metadata fields is a big problem. We currently have untrustworthy repositories full of software that would benefit from better description. Whatever model of package distribution you choose the metadata I have described is useful, excluding the signed md5. Probably none of it is essential, however, if I have gone to 8 hours effort to compile a particularly troublesome application I don't think the cost of ticking a few boxes and writing (or copy/pasting) 100 words is prohibitive. As to whether contributors will bother to sign their packages, I think this may become more important to the situations in which Puppy will be used commercially. I think Mark has already said that he will need to establish a secure repository for minisys use. If you need people to trust that the software is vetted by you then sign it.

Whatever system used will definitely need to handle the existing packages we have, but these must be marked as untrusted and due to the lack of metadata they will probably not be so easy to search for .

This leads me to add
16) known dependencies (I'm not sure how you format this or if it just a freeform list)
17) software version (should allow author to submit new versions of existing packages which will mask the old ones unless they are specifically searched for)

[wiak]

Digitally signed packages could avoid this as there is no way that the mirror can be accused of interfering with the package.

I'm all for this digitally signing packages, but it has to be easy to do. Though I always create md5sums for anything I do, I haven't been digitally signing them, and wouldn't off the top of my head know how. I do understand the process well enough, and could find out how to do it easily enough I'm sure, but that is an extra effort. In other words, it would be good if some system were provided to automate or make easy such practices. Then there is more chance of such measures being used by those packaging new applications for Puppy in general. I've seen cases of people producing packages but not knowing how to generate an md5sum for them even ...

[wiak]

I also think that we need a better repository/package manager. PSI is wonderful, but could use some refinement. PetGet is absolutely atrocious when compared to the package managers of other distros.

I don't want to over-praise any alternative small Linux distribution on a Puppy forum, but I do suggest that it might be worth closely examining the way Slitaz does its package management, and indeed, the way it organises its overall website in general:

http://www.slitaz.org/en/doc/handbook/packages.html

It is, I feel, very impressive and professional.
Inspires confidence, I'd say. Something to aim for.

Note that Slitaz also appears to be multi-user. Nathan F. expressed some of the security benefits multi-user capability well, a couple of posts down the following thread:

http://www.murga-linux.com/puppy/viewtopic.php?t=15839

I do hope the official Puppy releases become multi-user one day too.
Slitaz is destined to shoot up Distrowatch rankings. I would hate Puppy to be left behind.

Note, however, that I think it is not correct to view other small distributions as being in competition. Rather, they follow similar anti-bloat philosophies and their users and developers can learn from each other.

[prit1]

@wiak: I had never heard of a distribution called Slitaz before. I must admit, I dont check distrowatch that often now that I am hooked onto Puppy. But I am looking forward to give Slitaz a try when I get a chance.

I agree that we need to make a central repository and the mirrors should follow the same structure as the main repository. Also the repository must be categorized into meaningful and easily understandable sections like:
Base System:
Office:
Multimedia:
Utilities:
System-Tools:
WindowManagers:
XServer
etc.

MU's PSI does categorize it to some extent. But we must come up with something more standard.

Does anyone else think we need a team for creating and maintaining such a standard repository for Puppy?

[prit1]

Just a proof of concept:

How does this sample repository look like?

http://hardkap.net/puppyrepo

Features:
- Search the repository based on the category name, filename, description
- Description field available - must be added
- Is organized in categories

[HairyWill]

nice
I've used this successfully to download individual packages and also to and download several in one zip.
Is the search just across the package names or do you have the categories associated with them?