Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 09 Dec 2019, 18:25
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
Validating Package Archives and Metadata
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [3 Posts]  
Author Message
s243a

Joined: 02 Sep 2014
Posts: 2199

PostPosted: Sun 24 Nov 2019, 03:59    Post subject:  Validating Package Archives and Metadata
Subject description: Some **Very Draft** Code!
 

This is related to a previous thread:

Using gpg/pgp signatures in Package Managers

The previous thread focused more on the how and why, we should use various tools to validate data. I would like here to flesh out some draft implementaiton details that I want to apply to my fork of Scotmann's package manager (aka package).

This code is currently untested because it is in a draft state but writing this will help me know where to continue tomorrow.

In the debian/apt sources.list one can specify extra options in square. For example one can have [1]:

Code:
deb [signed-by=6809F110790E0720856B562F744ACF4DF3319FFA] https://deriv.example.net/debian/ stable main

https://wiki.debian.org/DebianRepository/UseThirdParty

In the above case the value assigned to signed-by is the pgp fingerprint of a public key. A fingerprint is generated by taking a cryptographic hash to the public key [2] and it uniquely identifies the public key. If you specify a fingerprint of the public key then unless you use the exclamation mark character (i.e. "!") all subkeys will be considered valid keys for signing [3] . My guess is a subkey is a key that is signed by said key and hasen't yet been revoked or expired.

As an alternative to specifying the fingerprint of the public key you can specify a path to a keyring. In this example rather than using the sources.list format we will use the DEB822-STYLE format [1][3]:

Code:

Types: deb deb-src
URIs: https://deriv.example.net/debian/
Suites: stable
Architectures: i386 amd64
Components: main
Signed-By: /usr/share/keyrings/deriv-archive-keyring.gpg

https://wiki.debian.org/DebianRepository/UseThirdParty

The DEB822-STYLE is not yet implemented in either sc0ttman's official version of pkg or my fork. In my fork I'm focusing on the sources.list style first.

So some preliminary untested code

The first thing we want to do is parse the options from the sources.list file. In the current official version of pkg, the extra sources.list options are stripped out:
Code:

    local apt_sources_list="$(grep -h '^deb ' /etc/apt/sources.list /etc/apt/sources.list.d/*.list 2>/dev/null \
      | sed \
        -e "s/^deb //g" \
        -e "s/^tor+//g" \
        -e 's/\[arch=[a-z,0-9].*\] //g' \
        -e 's/ /|/g'\
    )"

https://gitlab.com/sc0ttj/Pkg/blob/master/usr/sbin/pkg#L1768

and also the code only assumes that there is at most one extra option (i.e. "arch"). However, there are actually quite a bit more possibilities [3] such as: Languages, Targets, PDiffs, By-Hash, Allow-Insecure, etc...

So first we must parse out these options before we strip them with something like
Code:

 "sed -e 's/\[arch=[a-z,0-9].*\] //g' \"

as is done in the official version of pkg.

The tor flag is okay to remove because we should be able to tell that we want to use tor by the .onion domain. In the official pkg code the empty space character is replaced with the pipe character so that the "for in" notation can be used. This avoid process substitution which isn't supported by ash. The official version of package uses ash but my fork uses bash.

In my draft code the options are parsed out as follows (draft untested code):

Code:

      if [ ${line:0:1} = "[" ]; then
        local ind=`expr index "$line" "]"`
        local ind2=$((ind - 1))
        local line_opts=${line:1:$ind2}
      else
        local line_opts=''
      fi

https://pastebin.com/9ECEhZng

This is somewhat redundant because in a function that gets called later this will be done if it hasn't been done already (more untested code):

Code:

parse_sources_list_opts(){
    local line="${1/#deb /}" #Remove
   
      if [ ${line:0:1} = "[" ]; then
        local ind=`expr index "$line" "]"`
        local ind2=$((ind - 1))
        local line=${line:1:$ind2}
      fi
   
    set -- $line
    for opt in "$@"; do
      opt="$(echo $opt | tr "-" "_")"
      echo "export SOURCES_LIST_OPT__$opt"
    done   
}

https://pastebin.com/9ECEhZng

What we are doing here is we are prepending SOURCES_LIST_OPT__ to these extra sources.list options and then exporting the variables so that they can be used in ppa2pup.

ppa2pup is called as follows:

Code:

        (
          if [ ! -z "$line_opts" ]; then
           eval parse_sources_list_opts "$line_opts"         
          fi
          ppa2pup ${line//|/ } 1>/dev/null && echo -e "${green}Success${endcolour}: Updated repo '$ppa_repo_name'"
        )

https://pastebin.com/9ECEhZng

The metadata for the repo is located at:
Code:

  URL=$(echo $1 | sed -e 's#/$##g')/dists/${distro_ver}/${repo_stream}/binary-${arch}/Packages.gz

https://pastebin.com/SCPpvejZ

Later in the code we will remove the last part of this url. The last part of the url is:
Code:

  ARCH_PATH="/binary-${arch}/Packages.gz"

https://pastebin.com/SCPpvejZ

and what we are left with is the URL to the directory where the metadata about the release is located:
Code:

[url]meta_url=${download_url//$ARCH_PATH/}[/url]

https://pastebin.com/SCPpvejZ

This either consists of two files:
Release and Release.gpg

Example: debian-stretch

or a single file which has the pgp signing information embedded in the file:
InRlease

Example: Tor-Devian-stretch

In my draft code I first try to download Release and Release.pgp and if this fails I try to download InRelease and then verify the signature:

Code:

   meta_download_failed=false
    wget --quiet "${meta_url}/Release" -O "$RELEASE_DIR/$repo_name/Release" 1>/dev/null && \
    wget --quiet"${meta_url}/Release.gpg" -O "$RELEASE_DIR/$repo_name/Release.gpg" 1>/dev/null || \
      meta_download_failed=true 
    if [ "$meta_download_failed" = false ]; then
      if ( cd $RELEASE_DIR/$repo_name;
           $PGP_VERIFY_CMD --keyring "$SOURCES_LIST_OPT__signed_by" Release.pgp Release ); then
         pgp_verified=maybe
         ReleaseFile="$RELEASE_DIR/$repo_name/InRelease"   
      else
         pgp_verified=false
      fi
    else
        meta_download_failed=false
        wget --quiet "${meta_url}/InRelease" -O "$RELEASE_DIR/$repo_name/InRelease" 1>/dev/null \
          || meta_download_failed=true       
       if [ "$meta_download_failed" = false ]; then
         if ( cd $RELEASE_DIR/$repo_name;
            $PGP_VERIFY_CMD --keyring "$SOURCES_LIST_OPT__signed_by" InRelease ); then
            pgp_verified=maybe
            ReleaseFile="$RELEASE_DIR/$repo_name/InRelease" 
         else
           pgp_verified=false
         fi       
       fi 
    fi
    if [ "$pgp_verified" = false ]; then 
      if [ ! -z "$SOURCES_LIST_OPT__Trusted" ] && [ "$SOURCES_LIST_OPT__Trusted" = yes ]; then
        FAIL_CLASS=WARNING
      else
        FAIL_CLASS=ERROR
      fi   
      echo
      echo "$FAIL_CLASS: could not verify pgp signature of Release/InRelease file"
      echo
      echo "  ${meta_url}"
      if [ ! "$SOURCES_LIST_OPT__Trusted" ]; then #Maybe should exit if this
        exit 1
      fi
    fi   

https://pastebin.com/SCPpvejZ

If the signature is valid then we check the hash of the metadata (i.e. Packages.gz)

Example Tor (Uncomressed): Packages

Note that I believe that the hash is of the uncompressed data (to verify in testing).

Anyway, once the hash is computed of the metadata then we look in the release file to see if we can find a matching hash. Extra options in sources.list specify how strong a hash is required:

Code:

    if [ SOURCES_LIST_OPT__Allow_Insecure = yes ]; then 
      hash_function=( sha256sum md5sum sha1sum none )
    elif [ "$SOURCES_LIST_OPT__allow_weak" = yes ]; then
      hash_function=( sha256sum md5sum sha1sum )
    else
      hash_function=( sha256sum )
    fi

https://pastebin.com/SCPpvejZ

We now try each hash starting from the strongest to the weakest (or none in the insecure case) and fail if we can't verify the metadata with a strong enough hash:

Code:

   for a_hash_fn in "${hash_function[@]}"; then
      case "$a_hash_fn" in
      sha256sum)
        fsum="$(sha256sum "$file_path" | cut -f1 -d' ')"
        if [ $(grep -m -c "$fsum" "$ReleaseFile" ) -gt 0 ]; then
           break
        fi ;;
      md5sum)
        fsum="$(md5sum "$file_path" | cut -f1 -d' ')"
        if [ $(grep -m -c "$fsum" "$ReleaseFile" ) -gt 0 ]; then
           echo "WARNING: weak checksum for $repo_name"
           break
        fi ;;
      sha1sum)
        fsum="$(sha1sum "$file_path" | cut -f1 -d' ')"
        if [ $(grep -m -c "$fsum" "$ReleaseFile" ) -gt 0 ]; then
           echo "WARNING: weak checksum for $repo_name"
           break
        fi ;;
      none)
        fsum="$(md5sum "$file_path" | cut -f1 -d' ')"
        echo "WARNING: repo metadata unverified"
        break ;;
    fi
  fi


Okay, that's an overview. Hopefully the testing won't be too painful.

One thing that I haven't tried to implement yet is looking inside the .dep file for signed hashes if I'm not able to get this info from the repo metadata. See:

https://blog.packagecloud.io/eng/2014/10/28/howto-gpg-sign-verify-deb-packages-apt-repositories/

this approach might not be as secure though. See:

https://www.chosenplaintext.ca/articles/signing-files-vs-signing-hashes.html

Notes
------------------
1 - https://wiki.debian.org/DebianRepository/UseThirdParty
2 - https://en.wikipedia.org/wiki/Public_key_fingerprint
3 - https://manpages.debian.org/buster/apt/sources.list.5.en.html

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2199

PostPosted: Mon 25 Nov 2019, 03:00    Post subject:  

I've added code that in theory should do what I want but I've only tested it without the certificate checking. This is I suppose a bit better than fixing syntax errors. Smile

Here are some recent applicable commits:


Commits on Nov 19, 2019
24f47a0 add md5 stuff in ppa2pup and fixes for use awk in ppa2pup. See: https…

Commits on Nov 21, 2019
730b209 add SHA256 code to ppa2pup and fix awk parser. See: https://gitlab.co…
5346005 add SHA256 support
Commits on Nov 24, 2019
13b93f8 In ppa2pup lookup distro in sources-all. Some prep for certificate ch…

FIrst I needed some tools to check the certificate. I first tried:

Code:

pkg --get gpgv_2.1.18-8~deb9u4

but I realized that this tool could only check certificates and I couldn't use it to locate keys. So I needed the full verison instead:
Code:

pkg --get "gnupg_2.1.18-8~deb9u4


So now it's time to do some testing. I thought I would test the pgp checking first with tor [1]. Following the instructions at:
https://support.torproject.org/tbb/how-to-verify-signature/

I get the following output:
Code:

# gpg --auto-key-locate nodefault,wkd --locate-keys torbrowser@torproject.
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created

In the next part of instructions the tor website says:
Code:

gpg --output ./tor.keyring --export 0xEF6E286DDA85EA2A4BA7DE684E2C6E8793298290


However, the following debian link:
https://wiki.debian.org/DebianRepository/UseThirdParty
suggests that I should save keyrings at:
/usr/share/keyrings/

Therefore, let's modify the above command as follows:
Code:

gpg --output /usr/share/keyrings/tor.keyring --export 0xEF6E286DDA85EA2A4BA7DE684E2C6E8793298290


Now let's modify /etc/apt/sources.list as follows:
Code:

deb  [signed-by=/usr/share/keyrings/tor.keyring]  https://deb.torproject.org/torproject.org stretch main


Now, we must test the code (with tracing as follows):

Code:

bash -x /usr/sbin/pkg --repo-update 2>&1 | tee pkg_repo_update


I will now test the code and post updates once I've finished troubleshooting.

Edit: There are two dependencies which pkg didn't seem to install. These are dirmngr, libassuan0, libksba8 and libnpth0. I'm still getting errors though:

Code:

# gpg --auto-key-locate nodefault,wkd --locate-keys torbrowser@torproject.org
gpg: connecting dirmngr at '/root/.gnupg/S.dirmngr' failed: IPC connect call failed
gpg: error retrieving 'torbrowser@torproject.org' via WKD: No dirmngr
gpg: error reading key: No dirmngr



Notes
-----------
1 - https://support.torproject.org/tbb/how-to-verify-signature/

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2199

PostPosted: Mon 25 Nov 2019, 09:13    Post subject:  

I added the extra dependencies noted in my last post (see edit) and now I get:
Code:

# gpg --auto-key-locate nodefault,wkd --locate-keys torbrowser@torproject.org
gpg: error retrieving 'torbrowser@torproject.org' via WKD: Address family not supported by protocol
gpg: error reading key: Address family not supported by protocol


One can fix this by adding "disable-ipv6" to dirmngr.conf. See:
https://unix.stackexchange.com/questions/473629/gpg-keyserver-receive-failed-address-family-not-supported-by-protocol

or alternatively on could enable ipv6 which can be done first with:

Code:

modprobe ipv6

Find your router ip address with the command [1]:
Code:

ip -6 neigh show dev wlan0

and then add a route to the internet via your router [2]:
Code:

route -A inet6 add default gw ROUTER_IPv6 address


I think I'll try the IPv6 approach first. Note if ones ISP doesn't support IPv6 they can use Teredo_tunneling. If that fails then I'll disable ipv6 in dirmngr. Perhaps I should do this anyway

edit: I forgot, that after doing the modpribe for IPv6 one should first reconnect to the internet before doing the following two commands above, otherwise when you do neighbour discovery for the router you might get a stale IP address.

Notes:
--------------
1 - http://www.tldp.org/HOWTO/html_single/Linux+IPv6-HOWTO/#chapter-Neighbor-Discovery
2 - https://www.tldp.org/HOWTO/html_single/Linux+IPv6-HOWTO/#idp57822432

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 1 of 1 [3 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0578s ][ Queries: 11 (0.0052s) ][ GZIP on ]