Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 19 Sep 2019, 21:33
All times are UTC - 4
 Forum index » Advanced Topics » Cutting edge
Pkg - CLI package manager
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 9 of 10 [139 Posts]   Goto page: Previous 1, 2, 3, ..., 7, 8, 9, 10 Next
Author Message
sc0ttman


Joined: 16 Sep 2009
Posts: 2700
Location: UK

PostPosted: Sun 23 Jun 2019, 12:36    Post subject:  

s243a wrote:
I noticed an error that is causing double slashes.

the variable whoami isn't defined until line 100
Code:

whoami=$(whoami)

/usr/sbin/pkg#L100

but this variable is used to define the tmp directory on line 40 before this variable is set:
Code:

TMPDIR=/tmp/pkg/${whoami}  # set the tmp dir

/usr/sbin/pkg#L40

Yes this is a genuine bug, probably from when I moved some code around..

It causes stuff to be written into /tmp/pkg/ instead of /tmp/pkg/root/ ... or something like that .. not fatal, but will fix it anyway..

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2700
Location: UK

PostPosted: Sun 23 Jun 2019, 12:38    Post subject:  

s243a wrote:
I eventually did get an error (prior to fixing distro specs). The error was:

Code:

   Failed to download db file:
    http://packages.devuan.org/merged/dists//main/binary-/Packages.xz
   ...exited from 0setup script.

so pkg could sand this log file for the word "Failed"

Will look into doing something like that if 0setup fails, thanks.

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2700
Location: UK

PostPosted: Sun 23 Jun 2019, 12:45    Post subject:  

s243a wrote:

As a side note, one can see the builtin packages with the command:
Code:

HIDE_BUILTINS=false pkg --list-installed | grep ^hicolor

^ all packages will be shown (builtins, devx files and also user installed packages).

Pkg should probably separate devx packages out, so they can be hidden/shown with:

Code:
HIDE_DEVX_PACKAGES=true/false pkg --list-installed

Is that useful? Pkg counts devx packages as installed by default, so they don't get re-downloaded & intstalled...

Quote:
**BTW although I was missing the petspec record for libc6
Where? You mean ~/.packages/**.files?
Quote:
in my opinion having the file list should be enough to prevent the package from being installed...but this is a minor point.

Happy for this to be the default Pkg behaviour, if it isn't already .. might look into on a standard Puppy.

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Sun 23 Jun 2019, 18:44    Post subject:  

sc0ttman wrote:
s243a wrote:
sc0ttman wrote:

Of course, I first and foremost intend to get Pkg working in official Puppies ...
Your devuan acsii version is an 'edge case', for now...

Not sure if the following is an "edge case issue" or a woof-CE issue

If you are not running a standard Puppy (which you are not), then for Pkg it's 100%
an "edge case"
- because Pkg is designed (mainly) for standard Puppy Linux releases.

I have no idea if your custom OS (half Puppy, half something else) has everything setup
correctly... I don't even know if it was built with Woof... Hence, your OS is the "edge-case"
as far as Pkg development is concerned.


I started a thread on it:
tiny_puduan_ascii-PreAlpha11 (made via a woof-next fork)


It's not built via woof but it uses puppy init scripts and most of the rootfs-skeleton.

Code:

Anyway, I tried to work out if it's a Pkg issue....

thankyou Smile

Quote:

Quote:
I know that the package manager...
Which one?? Pkg? PPM? PPM 3.0?
Quote:
..writes to the actual save layer but in my case
the save layer is:

Code:

/initrd/mnt/dev_save


.....

Edit: After further examination this might be an issue with mistfire's package manager.

I search for pup_ro1 returns nothing that seems relevant in the gitlab branch for pkg.

Then I think you should probably post it at the PPM 3.0 thread, or even better work out if it's
actually a Woof-CE issue before posting at either... I guess..?

Quote:

I did notice though that pup_r01 is used in sfs_loadr.

Code:

   6) SAVE_LAYER='/pup_rw'; PUP_HOME='/pup_rw'; PUPSAVE='sda1,ext2,/';;
   12) SAVE_LAYER='/pup_rw'; PUP_HOME='/mnt/dev_save';;
   13) SAVE_LAYER='/pup_ro1'; PUP_HOME='/mnt/dev_save';;
   77) SAVE_LAYER='/pup_ro1'; PUP_HOME=''; PUPSAVE='sr0,iso9660,/2011-01-27-20-26';;

/usr/sbin/sfs_loadr#L1697

....

I think that PUP_HOME should either be /mnt/home or /initrd/mnt/dev_save/

**note that /mnt/home is a symlink pointing to /initrd/mnt/dev_save

Yep.. That is what I have on my system (Stretch RC something..) and most Pups I'd imagine..
But it depends on the install method/type... Sorry but I can't reproduce those issues..

...More generally ...

I think maybe you're posting "too soon", if you don't mind me saying so...

The first long series of posts you posted to the Pkg thread turned out to be your /etc/DISTRO_SPECS
not being setup correctly, which wouldn't have happened in a standard Puppy...


In my opinion pkg should in theory still work with an incorrectly configured /etc/DISTRO_SPECS, because there is enough information in /root/.pkg/sources to find the DB file online for the package. For instance in my setup I have the following (for the 1st line):
Code:

ascii-main|deb|Packages-devuan-ascii-main|http://deb.devuan.org/merged/||||noarch common ascii ascii-backports ascii-contrib ascii-multimedia ascii-non-free stretch-main stretch-backports stretch-contrib stretch-multimedia dpup upup


The path to the DB file is:
Code:

http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.gz   
http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.xz   


the only part of these paths not in souces is the part that says "dist" and the part that says "binary-i386". In lieu of this info pkg could try testing alternatives and could ask the user if they want to over-ride the settings in distro specs.

In my case initially pkg would fail because I originally had TARGET_ARCH='x86' when it should be TARGET_ARCH='i386' but if pkg was able to infer the architecture (e.g. looking at the elf type of libc) then it would know to try i386 in the path.
**maybe a file could be added which specifies the "ARCH" format for each repo.

Note that at the moment some of this is outside the scope of "pkg" because the repo update is done by files which are currently part of puppies package manager.

Code:
And sorry, but I also don't want this thread mixed up with mistfires PPM 3.0 (which literally runs
[i]against[/i] to the goals of this project).. Pkg is not designed to support PPM 3.0 (unless it becomes
Puppys official/main PPM).. even then, Pkg aims to become independent of any PPM..

...

I think you should probably test Pkg on a standard Puppy to compare to your OS before posting any
issues here in order to make sure the issues are with Pkg and not your custom OS..


Currently pkg still depends on some files from the puppy package manager. These files could be from the current puppy package manager, mistfires version or the latest woof-CE code. Since pkg aims to be independent of the ppm, may I suggest having a variable that points to the directory where the needed ppm files are. One could make this independent of puppies ppm by simply pointing to a different directory than the default puppy ppm.
**although it might also want to use a different directory for temporary files.

Anyway, regarding testing on a standard puppy; it seems to be the case that there are a lot of woof-CE changes related to puppies package manager. Some of these are related to improvements by mistfire. If pkg aims to be compatible with the lastest woof-CE code then the code is too unstable at this point. If pkg aims to use legacy puppy package manager code then it should include these files as part of the project and put them in a separate directory than the default puppy package manager.

Anyway, the reason that mistfire's code interested me was that in her x-slacko slim release the ppm appeared to be able to downloaded packages from many different repos.
**In theory pkg should also be able to do this based on the soures file.

For this to work, I think that the dependencies on DISTRO_SPECS should be removed, in mistfire's ppm 3.0 thread, it appears to me that there are still some dependencies on DISTRO_SPECS related to updating the package data base files. Therefore, I'm not sure if x-slacko slim properly updates databases which aren't part of the compatible distro.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2700
Location: UK

PostPosted: Mon 24 Jun 2019, 14:17    Post subject:  

s243a wrote:
In my opinion pkg should in theory still work with an incorrectly configured /etc/DISTRO_SPECS, because there is enough information in /root/.pkg/sources to find the DB file online for the package. For instance in my setup I have the following (for the 1st line):
Code:

ascii-main|deb|Packages-devuan-ascii-main|http://deb.devuan.org/merged/||||noarch common ascii ascii-backports ascii-contrib ascii-multimedia ascii-non-free stretch-main stretch-backports stretch-contrib stretch-multimedia dpup upup


The path to the DB file is:
Code:

http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.gz   
http://deb.devuan.org/merged/dists/ascii/main/binary-i386/Packages.xz   


the only part of these paths not in souces is the part that says "dist" and the part that says "binary-i386". In lieu of this info pkg could try testing alternatives and could ask the user if they want to over-ride the settings in distro specs.

Ah OK, that may be another area where we can stop Pkg relying on Puppy config files... Will have a look into it at some point..

Quote:
In my case initially pkg would fail because I originally had TARGET_ARCH='x86' when it should be TARGET_ARCH='i386' but if pkg was able to infer the architecture (e.g. looking at the elf type of libc) then it would know to try i386 in the path.
**maybe a file could be added which specifies the "ARCH" format for each repo.

It already does do some of its own arch checking, so should be do-able..

As for Pkg/Woof-CE compatibility - yes there is a way to go yet...

And the only trouble I see with using Pkg alongside PPM 3.0 is that I've no idea what changes were made - except I do know some of the ~/.package/ files have been re-organised ... Obvioulsy if you can make Pkg work alongside PPM 3.0 that is great, but I won't be bug-fixing Pkg to make it happen ... Only to align it with whatever PPM is in Woof-CE ..

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Tue 25 Jun 2019, 23:32    Post subject:  

I'm not completly sure if the following is a bug or not but I noticed that you do the "direct save path" slightly different than puppy:

Code:

# set $DIRECTSAVEPATH (where we want to install pkgs)
if [ $PUPMODE -eq 3 -o $PUPMODE -eq 7 -o $PUPMODE -eq 13 ];then
  DIRECTSAVEPATH="/initrd${SAVE_LAYER}" #SAVE_LAYER is in /etc/rc.d/PUPSTATE.
elif [ "$PUPMODE" = "2" ]; then
  DIRECTSAVEPATH=""
fi

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L147

Code:

      if [ ! "$DIRLINK" ];then
       if [ -h "/initrd${SAVE_LAYER}${ONESPEC}" ];then #120107
        DIRLINK="`readlink -m "/initrd${SAVE_LAYER}${ONESPEC}" | sed -e "s%/initrd${SAVE_LAYER}%%"`" #SAVE_LAYER: see /etc/rc.d/PUPSTATE. 120107
        xDIRLINK="`readlink "/initrd${SAVE_LAYER}${ONESPEC}"`" #120107
       fi
      fi
      if [ "$DIRLINK" ];then
       if [ -d "$DIRLINK"  ];then
        if [ "$DIRLINK" != "${ONESPEC}" ];then #precaution.
         mkdir -p "${DIRECTSAVEPATH}${DIRLINK}" #120107
         cp -a -f --remove-destination ${DIRECTSAVEPATH}"${ONESPEC}"/* "${DIRECTSAVEPATH}${DIRLINK}/" #ha! fails if put double-quotes around entire expression.
         rm -rf "${DIRECTSAVEPATH}${ONESPEC}"
         if [ "$DIRECTSAVEPATH" = "" ];then
          ln -s "$xDIRLINK" "${ONESPEC}"
         else
          DSOPATH="`dirname "${DIRECTSAVEPATH}${ONESPEC}"`"
          DSOBASE="`basename "${DIRECTSAVEPATH}${ONESPEC}"`"
          rm -f "${DSOPATH}/.wh.${DSOBASE}" #allow underlying symlink to become visible on top.
         fi
fi

/woof-code/rootfs-skeleton/usr/local/petget/installpkg.sh#L443

The key difference here is that in puppy they are using the "readlink" function, so the puppy package manager gets the real path whereas you are using a symlink which points to the "direct save path" The consequence is that when you you extrat the package archive via the tar command, you are extracting the package archive to the symlink rather than to the actual directory.

Code:

tar --absolute-names $tarops "./${PKGNAME}.tar.$TAREXT" ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog

/master/usr/sbin/pkg#L4563

Now long ago (perhaps prior to when woof-CE was on github there was the following fix:
Code:

#120102 install may have overwritten a symlink-to-dir.
#120107 rerwin: need quotes around some paths in case of space chars. remove '--unlink-first' from tar (was introduced 120102, don't think necessary).

woof-code/rootfs-skeleton/usr/local/petget/installpkg.sh#L26

I'm guessing that the --unlink-first option was used in conjuction with the "-h" option:
Quote:

-h, --dereference
follow symlinks; archive and dump the files they point to

http://murga-linux.com/puppy/viewtopic.php?p=1030949#1030949

However, I assume that neither of these options are necessary now that the puppy package manager is using the realpath of pup_ro1

Anyway, this error only manifested itself (see post) when I broke either "busybox mount" or mount.aufs (the latter used in mount-FULL) and therefore, I don't know yet whether or not this is an actual bug in "pkg".

Edit:

The following section of the "tar manpage" suggests that /initrd/pup_ro1 will be overwritten if the following option isn't supplied (testing required to verify):

Quote:

--keep-directory-symlink
Don't replace existing symlinks to directories when
extracting.

http://man7.org/linux/man-pages/man1/tar.1.html

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Wed 26 Jun 2019, 12:33    Post subject:  

sc0ttman wrote:
s243a wrote:
In the meantime I took a stab at this (commit e4b000bda57f13a6eba5a2f3fb2bd43e856189ce)

I probably made some changes which weren't necessary.


You should probably define the right package dir at the top of the /usr/sbin/pkg script as a variable,
then use that variable later throughout the script..

Or even better, define it in the main config RC script: ~/.pkg/pkgrc

Then it can be set to various different places - easier to run Pkg in chroot, ssh, local folder for unit tests Wink , etc..


Okay, I took your suggestion (see commit: ea299b8ff3054173df6b5725bc8b7eef658762c6). I put the following

Code:

[ -z "$PKGS_DIR" ] && PKGS_DIR="$(realpath "${HOME}/.packages")"

/woof-code/rootfs-packages/PKG/usr/sbin/pkg#L60

after "~/.pkg/pkgrc" is sourced. Therefore this folder can be defined in ~/.pkg/pkgrc.

I left in the previous realpath statements I added. I don't think they hurt anything but are probably redundant.

**Note that this last change was done by a simple string replace and I haven't done any testing yet.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Wed 26 Jun 2019, 21:14    Post subject:  

s243a wrote:
sc0ttman wrote:
s243a wrote:
I'm not sure if the following is being done by ppm v3.0 or sc0tmann's pkg but some process seems to be moving all my files out of the /usr/lib/i386-linux-gnu/ folder....

...This time the whole i386-linux-gnu folder seems to be replaced with a symlink. I know this is standard puppy to do this but maybe there is a reason that Debian/Devaun creates the i386-linux-gnu folder (e.g. to separate architectures on multi architecture systems).

Maybe pkg and PPM..? Don't quote me on that..

But this code in Pkg expects a symlink:



Your are correct. This is related to the "muti-arch hack" of the puppy package manager. I oppened an issue on github to discuss whether said muti-arch hack is a good or bad thing:

https://github.com/puppylinux-woof-CE/woof-CE/issues/1475

http://murga-linux.com/puppy/viewtopic.php?p=1030959#1030959

I found the code that sc0tmann is talking about here:

Code:


               # woofce: NO_MULTIARCH_SYMLINK=1 (DISTRO_SPECS)
               if [ -z "$NO_MULTIARCH_SYMLINK" ] ; then
            if [ -f $HOME/.packages/${PKGNAME}.files ];then
              pkg_has_archdir="$(grep -m1 "$DISTRO_ARCHDIR" "$HOME/.packages/${PKGNAME}.files")"
            fi

                  # Workaround to avoid overwriting the $DISTRO_ARCHDIR symlink.
                  if [ "$DISTRO_ARCHDIR" != "" -a -f $HOME/.packages/${PKGNAME}.files -a "$pkg_has_archdir" != "" ]; then
                    mkdir -p /tmp/$PKGNAME
                    rm -rf /tmp/$PKGNAME/*
                    dpkg-deb -x ${CURDIR}/${PKGNAME}.deb /tmp/$PKGNAME/ 2> $TMPDIR/$SELF-cp-errlog
                    for f in $(find /tmp/$PKGNAME \( -type f -o -type l \))
                    do
                       xpath=$(echo "$f" |  cut  -f 4-30 -d "/" | sed "s/$DISTRO_ARCHDIR\///")
                       mkdir -p ${DIRECTSAVEPATH}/$(dirname "$xpath")
                       cp -a "$f" ${DIRECTSAVEPATH}/$(dirname "$xpath")/
                    done
                    rm -rf /tmp/$PKGNAME &>/dev/null
                  else
                    dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
                  fi
                else
                  dpkg-deb -x ${CURDIR}/${PKGNAME}.deb ${DIRECTSAVEPATH}/ 2> $TMPDIR/$SELF-cp-errlog
                fi

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L4614

BTW: according to wdlkmpx, the "multi-arch hack" has been removed:
https://github.com/puppylinux-woof-CE/woof-CE/issues/1475#issuecomment-505873406

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Sat 20 Jul 2019, 20:29    Post subject:  

The following doesn't look correct to me:
Code:

dpkg-deb -e "${CURDIR}/${PKGNAME}/${PKGNAME}.deb" /DEBIAN #130112 extracts deb control files to dir /DEBIAN. may have a post-install script, see below.

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L4644

I think it should be:
Code:

dpkg-deb -e "${CURDIR}/${PKGNAME}.deb" /DEBIAN #130112 extracts deb control files to dir /DEBIAN. may have a post-install script, see below.


It also looks like it is code that was copied from woof-CE so there may have been (or be) a similar bug in woof-CE.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2700
Location: UK

PostPosted: Sun 28 Jul 2019, 05:47    Post subject: pkg bug found  

Something good:

I (stupidly) only just realised there is an easy way to get updated packages.

And it will get around the horrible merging of repos that the PPM insists on
doing during repo updates...

How?

You can manually add the "backports" or "unstable" (etc) repos to Puppy
as separate repos, by doing the following:

Code:
pkg add-repo http://deb.debian.org/debian stretch-backports main contrib


^ this makes it possible to add extra repos (updates, backports or testing) to your
Ubuntu/Debian based Puppies very easily..

After adding the backports repo to my Puppy Stretch 7.5 RC2, I now have
a user-added repo (called stretch-backports) that has lots of updated versions of the
packages in my main repos Smile - something the default Puppy repo update (`0setup`)
fails to do (for some reason!)...


Something bad:

There is a serious bug in the `add-repo` command:
Sometimes the newly added repo overwrites another repo!


Details:

If you add a 3rd-party Debian repo that has streams matching your default
system repos, then the new repo will replace the system one whose stream
names it matches, wiping out the main repo in the process.

Example which breaks the repos:

Code:
pkg add-repo https://deb.torproject.org/torproject.org stretch main


^this doesn't work cos the last 2 fields (the repo "streams") are "stretch" and "main",
so the "stretch-main" repo gets overritten!!

Workaround for now:

Install tor from the Ubuntu PPA repos instead:

Code:
pkg add-repo http://deb.torproject.org/torproject.org/ bionic main

_________________
Pkg, mdsh, Woofy, Akita Linux, VLC-GTK
Back to top
View user's profile Send private message 
josejp2424


Joined: 01 Aug 2010
Posts: 492

PostPosted: Sun 28 Jul 2019, 09:13    Post subject: thanks for this wonderful pkg-cli !!!
Subject description: thanks for this wonderful pkg-cli !!!
 

I love this program in my case it works very well. Thank you sc0ttman.
and I don't report an error, because I didn't find them. Wink


_________________
Shiba Inu | Pupjibaro jessie | My Blog
Back to top
View user's profile Send private message Visit poster's website 
josejp2424


Joined: 01 Aug 2010
Posts: 492

PostPosted: Sun 28 Jul 2019, 20:42    Post subject: DpupBuster pkg
Subject description: DpupBuster pkg
 

http://murga-linux.com/puppy/viewtopic.php?p=1033179#1033179

sc0ttman wrote:
josejp2424 wrote:
pkg seems to be working well now

Have you added any fixes to Pkg to make it work?

What was the problem, if I may ask?
.

hello sc0ttman.
Regarding this question.
in the previous version woof-ce. the x86_64-linux-gnu folders.
They had link to / lib. and pkg when installing packages modified the folder. I stopped having a / lib link.
And corrupted the system.

in the new version of woof-ce.
those folders no longer have a direct link to / lib.


http://murga-linux.com/puppy/viewtopic.php?t=116615

_________________
Shiba Inu | Pupjibaro jessie | My Blog
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Thu 22 Aug 2019, 20:09    Post subject:  

I think I know why pkg seems to be installing [1] some items which are already installed.

Pkg removes dependencies from the list of packages to install with the following line:
Code:

 comm -23 ${TMPDIR}all_deps_0 ${TMPDIR}installed_pkgs | sort -u | grep -v ^$ > ${TMPDIR}all_deps_1

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L6033

The listof files ${TMPDIR}installed_pkgs is produced by the following function:

Code:

list_all_installed_pkgs(){        # list inc builtins if HIDE_BUILTINS=false
  # reset list of installed pkgs
  echo -n '' > $TMPDIR/installed_pkgs

  if [ "${HIDE_INSTALLED}" = true -a "$FORCE" = false ];then
    # add user installed pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/user-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ "${HIDE_BUILTINS}" = true ];then
    # add builtins to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/woof-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/devx-installed-packages ];then
    # add devx pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/devx-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/layers-installed-packages ];then
    # add layers list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/layers-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $TMPDIR/installed_pkgs -a ! -z $TMPDIR/installed_pkgs ];then
    sort -u $TMPDIR/installed_pkgs | grep -v ^$ > $TMPDIR/installed_pkgs__sorted
    mv $TMPDIR/installed_pkgs__sorted $TMPDIR/installed_pkgs
  fi
}

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L5947

I think in the first to if blocks the logic is backwards but I'm not sure why we have the iff blocks at all. Why not just do the following instead:

Code:

list_all_installed_pkgs(){        # list inc builtins if HIDE_BUILTINS=false
  # reset list of installed pkgs
  echo -n '' > $TMPDIR/installed_pkgs

    cut -f2 -d'|' "$USER_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs
    cut -f2 -d'|' "$WOOF_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs
    cut -f2 -d'|' "$DEVX_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs
    cut -f2 -d'|' "$LAYER_INST_PKGS_FILE" >> $TMPDIR/installed_pkgs

  if [ -f $TMPDIR/installed_pkgs -a ! -z $TMPDIR/installed_pkgs ];then
    sort -u $TMPDIR/installed_pkgs | grep -v ^$ > $TMPDIR/installed_pkgs__sorted
    mv $TMPDIR/installed_pkgs__sorted $TMPDIR/installed_pkgs
  fi
}


For now I'll just comment out code to effectively do this in case I misunderstood something.

Notes
--------------------
I say "seems to be installing" because some of the packages that were installed as dependencies look like built-ins to me. I haven't verified this yet though.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Thu 22 Aug 2019, 21:07    Post subject:  

s243a wrote:
I think I know why pkg seems to be installing [1] some items which are already installed.
...
Notes
--------------------
I say "seems to be installing" because some of the packages that were installed as dependencies look like built-ins to me. I haven't verified this yet though.


I looked into this further. I tested this on tiny_puduan_ascii-PreAlpha11.4.iso (see post). I have the builtin file libstdc++6-6.3.0-18+deb9u1. The following package was installed (despite my above mods) libstdc++-5.0.6 (as part of installing dependencies for firefox-esr). Maybe this is because it is a different version number? I'll look into it more.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2073

PostPosted: Sat 24 Aug 2019, 09:41    Post subject:  

s243a wrote:
I
Pkg removes dependencies from the list of packages to install with the following line:
Code:

 comm -23 ${TMPDIR}all_deps_0 ${TMPDIR}installed_pkgs | sort -u | grep -v ^$ > ${TMPDIR}all_deps_1

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L6033

The listof files ${TMPDIR}installed_pkgs is produced by the following function:

Code:

list_all_installed_pkgs(){        # list inc builtins if HIDE_BUILTINS=false
  # reset list of installed pkgs
  echo -n '' > $TMPDIR/installed_pkgs

  if [ "${HIDE_INSTALLED}" = true -a "$FORCE" = false ];then
    # add user installed pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/user-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ "${HIDE_BUILTINS}" = true ];then
    # add builtins to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/woof-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/devx-installed-packages ];then
    # add devx pkgs to list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/devx-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $HOME/.packages/layers-installed-packages ];then
    # add layers list of pkgs to remove from final output
    cut -f2 -d'|' $HOME/.packages/layers-installed-packages >> $TMPDIR/installed_pkgs
  fi

  if [ -f $TMPDIR/installed_pkgs -a ! -z $TMPDIR/installed_pkgs ];then
    sort -u $TMPDIR/installed_pkgs | grep -v ^$ > $TMPDIR/installed_pkgs__sorted
    mv $TMPDIR/installed_pkgs__sorted $TMPDIR/installed_pkgs
  fi
}

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L5947


Trying to trace what is going on here might be slightly difficult due to concurancy but maybe I'll make good progress in understaning this by writing this post.

I noticed the currency via line L6311 of the following code:
Code:

5984 get_deps(){
...
5990 list_all_installed_pkgs
...
6311  # wait until list_deps() is finished (if it's running at all...)
6312  while [ -f /tmp/pkg/list_deps_busy ];do
6313    echo -n '.'
6314    sleep 0.75
6315  done

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L6311

My debug output execuded via
Code:

bash -x /usr/sbin/pkg 2>$1 | tee logfile


produces output like:
Code:

+ list_all_installed_pkgs
+ echo -n ''
+ cut -f2 '-d|' /var/packages/user-installed-packages
++ get_pkg_ext firefox-esr_60.6.3esr-1
++ '[' '!' firefox-esr_60.6.3esr-1 ']'
++ local pkg_installed=
++ local pkg_filename=
++ local pkg_ext=
++ local pkg_name=
++ case "$1" in
+ cut -f2 '-d|' /var/packages/woof-installed-packages
+++ grep -m1 '^firefox-esr_60.6.3esr-1'
+ cut -f2 '-d|' /var/packages/devx-only-installed-packages
+++ list_installed_pkgs firefox-esr_60.6.3esr-1
+++ local user_pkgs_list=
+++ local builtins_list=
+++ local devx_pkgs_list=
+++ user_pkgs_list=/var/packages/user-installed-packages
+++ '[' true '!=' true -a -f /var/packages/devx-only-installed-packages ']'
+++ '[' true '!=' true ']'
+++ '[' '!' firefox-esr_60.6.3esr-1 ']'
cut: /var/packages/devx-only-installed-packages: No such file or directory
+ cut -f2 '-d|' '/var/packages/layers-installed packages'
+++ grep '^firefox-esr_60.6.3esr-1'
+++ cut -f1 '-d|' /var/packages/user-installed-packages


What seems to be happening based on this output is that get_deps() is getting called multiple times in the background before the function list_all_installed_pkgs() finishes. "+" is the first instance "++" is the second instance and "+++" is the third instance of get_deps. The flag:
Code:

/tmp/pkg/list_deps_busy

seems to be used to prevent multiple instances of get_deps from calling list_all_installed_pkgs()


Note the output is from my modified version of pkg but the code that I'm citing is from sc0tmann's version. I'm speculating a bit here about concurrency because I haven't yet identified the code that launches the background processes.

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 9 of 10 [139 Posts]   Goto page: Previous 1, 2, 3, ..., 7, 8, 9, 10 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Cutting edge
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1299s ][ Queries: 12 (0.0239s) ][ GZIP on ]