packdude

Under development: PCMCIA, wireless, etc.
Message
Author
jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#46 Post by jamesbond »

Which repo URL can I use for testing packdude?
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#47 Post by Iguleder »

[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#48 Post by jamesbond »

Thanks.

Is there a way to download the packages for offline use? When building a distro, I don't want to keep re-downloading the packages over and over again.
These are the steps:
1) Get to know the list of available packages
2) Get a list of dependency for all packages.
3) I have a subset of packages I want to use (I may not want to install everything that exist in the repo). With information from 1) and 2), I want to download packages for offline storage (download only - don't install).
4) Install to chroot, with packages from local filesystem
- if it does not use dependency, then its fine.
- if it does use dependency, I want that dependency to be resolved from my local filesystem.

I think packdude can only do step 1 at the moment (and may be partially 4, as you said before).

For testing, I did this before:

Code: Select all

mkdir -p /root/p/var/packdude
export REPO=your-repo-url above.
for p in $(./packdude -p /root/p -l | sed 's/|.*//'); do ./packdude -p /root/p -i $p; done
And that gives me a nice filesystem, but if I want to do it again I end up re-downloading everything again.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
neerajkolte
Posts: 516
Joined: Mon 10 Feb 2014, 07:05
Location: Pune, India.

#49 Post by neerajkolte »

Hi jamesbond,

In which puppy you're testing Packdude.
In Fatdog-64, or the new puppy you're building with pure ubuntu repo packages using Fatdog's kernal.

If that's the case could you provide me link to libarchive for fatdog64-631.

I tried using Packdude but it asks me libarchive (see Here) . I couldn't find it.
I have no knowledge of compiling.
But i downloaded source from HERE. I don't even know if that's right.
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson

“We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.â€￾
- Amara’s Law.

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#50 Post by Iguleder »

jamesbond - you can download the packages and repo.sqlite3, using wget. All files are located at the repository root. Then, use file:// or python -m SimpleHTTPServer.
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#51 Post by jamesbond »

Iguleder wrote:jamesbond - you can download the packages and repo.sqlite3, using wget. All files are located at the repository root. Then, use file:// or python -m SimpleHTTPServer.
I visited your repo URL (using browser) and I saw nothing. And how would I know the package filenames to download? "packdude -l" doesn't show me the filenames.

Neerajkolte - I'm using Fatdog631. Indeed you need to build libarchive before you can build packdude. I have a pet for it - which I can upload, no problem - but no offense, are you sure you know what you're doing? This is a package manager, you can easily install files overwriting Fatdog's one which will screw up your installation. If you want to test it, the packages must be installed in a chroot directory (by default packdude will install its packages to your REAL ROOT and thus clobbering your setup).
Attachments
packdude-630.pet
For Fatdog64
(67.74 KiB) Downloaded 159 times
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#52 Post by Iguleder »

The repository is security hardened. You can see package file names during their download.
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#53 Post by jamesbond »

Code: Select all

sqlite3 repo.sqlite3 "select file_name from packages"
is a better way to do it. That solves point 2) and 3) from my original question.

More questions:
4. If I install package with dependency, say, mtpaint, from filesystem (file:///path/to/mtpaint.dude), what happens with its dependencies? Will it fetch its dependencies from the same directory on the filesystem, or will it attempt to fetch from the remote repo?

5. Alternatively, will REPO=file:///xxx work? Since you linked it libcurl I suppose it does, doesn't it?
EDIT: This is confirmed to work. So question no 4 is not relevant anymore.

While we're at it, why not add md5sum to the package database? Helps to ensure that downloaded packages aren't corrupted (and helps to avoid downloading when there is already a local copy and server's copy have not changed).

cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#54 Post by Iguleder »

Dependencies are fetched from REPO as well.

The package format has a checksum, so putting it in the database is a waste. Look at package.c.
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#55 Post by jamesbond »

Iguleder wrote:Dependencies are fetched from REPO as well.

The package format has a checksum, so putting it in the database is a waste. Look at package.c.
Okay, that solves the integrity problem. But I'd like to check for that fact before I run "packdude -i" (if you have packdude --check-integrity that returns true or false, that would be good enough).

The md5sum in the database serves a different purpose. It avoids unnecessary downloading when the server's copy of the package hasn't changed.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#56 Post by Iguleder »

packdude doesn't keep downloaded packages - the whole installation is done in-memory. Local package cache wastes save file space and it's good only if you re-install a package (a rare situation).
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#57 Post by jamesbond »

Well, that rare use case is what I've said before: when building a distro. I want to keep all the packages local so I don't have to re-download every time when I re-build the distro. And keeping the packages local means I need a means to ensure my local copy is updated, without the need to re-download every single package and comparing them.

Aside from that, as a user, I also like to keep a copy of the packages locally, in case I need that packages when the server's copy is gone (keeping old version of the packages -- since you say packdude can only keep one version of a package at one time, or in case I need to re-install when I'm offline, or when sharing packages with others nearby, or when the repo maintainer gets bored of maintaining it and kills the server :wink: ).

Please consider it.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#58 Post by Iguleder »

I can add a repository cloning mode, which downloads everything. What do you think?
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#59 Post by jamesbond »

Clone meets my needs, but it sounds like you're going to re-implement rsync - are you sure you want to do that?

1) How about adding a filter to the clone? fnmatch or regex would be fine. I want to clone but I don't want to necessarily copy everything (especially the repo grows big).

2) When I re-run clone over existing local repo, I assume it will only download changed or missing packages?
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#60 Post by Iguleder »

I think full cloning is a good start. Updating sounds like something I'd leave for LFTP and rsync.
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#61 Post by jamesbond »

Iguleder wrote:I think full cloning is a good start. Updating sounds like something I'd leave for LFTP and rsync.
No, then it won't help. I can already clone now (if I want to do partial cloning I just insert grep in between sqlite3 and xargs):

Code: Select all

sqlite3 /tmp/repo.sqlite3 "select file_name from packages" | xargs -I '{}' wget -c $REPO_URL/'{}'
It's the update which is a problem if I can't check that existing packages matches packages listed in new repo.sqlite3. I can in theory check using version numbers but sometimes the same package gets recompiled without changes in their version numbers - how do you deal with that? (Slackware and Debian packages have "build-numbers" that are supposed to deal with this case, but that isn't foolproof because it requires the packager to update build numbers each time he/she rebuild the package. Checksums are generated automatically each time the repo is updated - so no chance to miss it - and both Debian/Slackware have that, too).

Leaving the update to LFTP and rsync assumes that the repo is accessible by FTP and by rsync - which is a *big* assumption.

I think the best return/effort ratio is still just to add md5sum column in the database, and make sure that repodude does md5sum when it updates the database.

But I'll leave the decision to you :)
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

User avatar
neerajkolte
Posts: 516
Joined: Mon 10 Feb 2014, 07:05
Location: Pune, India.

#62 Post by neerajkolte »

jamesbond wrote:are you sure you know what you're doing?
Frankly NO :wink: . I was attracted towards packdude by
Iguleder wrote:Support for Debian, Slackware or Ubuntu packages can be provided in the form of converted packages (i.e a tool that converts an entire Slackware repository into a "native" repository).
I am not in pure sense "TESTING", merely just keeping up with the development. I feel packdude has potential to go long way.
jamesbond wrote:This is a package manager, you can easily install files overwriting Fatdog's one which will screw up your installation. If you want to test it, the packages must be installed in a chroot directory (by default packdude will install its packages to your REAL ROOT and thus clobbering your setup).
Thanks for the heads up. I also read this..
Iguleder wrote:packdude is still considered experimental and may ruin your system. Use it at your own risk.

You can try out packdude without doing any damage to your system, by using an installation prefix:

Code: Select all

mkdir -p /tmp/var/packdude/archive
packdude -p /tmp -i mtpaint
In this example, packdude installs mtPaint to /tmp. Before you run packdude with "-p", make sure you create the specified directory and /var/packdude/archive under it.
Personally I have no qualms about breaking sysyem. The only things I consider important and keep backups of are collection of videos and photos of friends and family.
Everything else is formattable or re-downloadable, especially the OS.
This is the thing that attracted me to puppy in the first place, ease of changing reconfiguring renewing the OS. The other thing being Friendly forum :D .
Thanks

-Neeraj
"One of my most productive days was throwing away 1000 lines of code."
- Ken Thompson

“We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.â€￾
- Amara’s Law.

Post Reply