Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Thu 27 Feb 2020, 14:14
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
awk: Converting deb dependency info into puppy format
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [4 Posts]  
Author Message
s243a

Joined: 02 Sep 2014
Posts: 2478

PostPosted: Tue 26 Nov 2019, 14:36    Post subject:  awk: Converting deb dependency info into puppy format
Subject description: applications to sc0ttman's package manager (i.e. pkg) and forks
 

Introduction

I'm convinced that awk based code will be orders of magnitude faster (see Issue #73) than the current ppa2pup code in "pkg", which is used in pkg to convert a debian style repo-database to the puppy format. Of course I need to verify my suspicions about these speed differences with actual testing.

This code (i.e. ppa2pup) in is only executed in pkg, if there is a corresponding entry in sources.list otherwise
/usr/local/petget/0setup
is used to convert the debian database code into puppy format.

The 0setup code might be comparable in speed to the awk code (need to verify) but it adds architecture dependencies to pkg and it is considerably harder to add a repo this way than by the sources.list approach. The 0setup code also removes the epoch from the version number, which I think it shouldn't.

Some Test Code

Code:

#!/bin/bash
distro=devuan
distro_ver=ascii
repo_filename="/var/packages/repo/Packages-devuan-ascii-main"
#a_stream=main
a_repo_url="http://deb.devuan.org/merged/"
         cat ~/awk_dep_test.in | awk -v PKGOS="$distro" -v PKGOSVER="$distro_ver" -v REPOFNM="$repo_filename" \
'function fixdepends_jamesbond(s,   a,p,sout) {
   split(s,a,",")
   for (p in a) {
      gsub(/[ \t]*\(.*\)|[ \t]\|.*|:any/,"",a[p])
      sout = sout "," a[p]
   }
   sub(/^,/,"",sout) ; return sout;
}
function get_inequality(s){
   switch(s){
      case "<":
        return "lt"
        break
      case ">":
        return "gt"
        break
      case "<=":
        return "le"
        break
      case ">=":
        return "ge"
        break
      case "=":
        return "eq"
        break

    }
    return ""
}
function fixdepends(s,   p,a,dep_i,dep_split,pkg_i,ver_i,v,ineq,sout) {
   split(s,a,",")
   for (p in a) {
      match(a[p],/^[ \t]*([^ \t]+)[ \t]*(\((.*)\))?/,dep_split)
      pkg_i = dep_split[1]
      sout = sout ",+" pkg_i
      if ( length(dep_split) > 1 ){
        ver_i=dep_split[3]
        split(ver_i,v," ")
        ineq=get_inequality(v[1])
        if ( length(v) > 1 ){
          sout = sout "&" ineq v[2]
        }
       }
   }
   sub(/^,/,"",sout) ; return sout;
}
/^Package:/     { sub(/^Package: /,"");  PKG=$0; }
/^Version:/     { sub(/^Version: /,"");  PKGVER=$0; }
/^Filename:/    { sub(/^Filename: /,""); PKGPATH=$0; sub(/\/[^\/]*$/,"",PKGPATH); sub(/.*\//,""); PKGFILE=$0; }
/^Priority:/    { sub(/^Priority: /,""); PKGPRIO=$0; }
/^Section:/     { sub(/^Section: /,"");  PKGSECTION=$0; }
/^Installed-Size:/ { sub(/^Installed-Size: /,"");  PKGSIZE=$0; }
/^MD5sum:/      { sub(/^MD5sum: /,"");   PKGMD5=$0; }
/^SHA1:/      { sub(/^SHA1: /,"");   SHA1=$0; }
/^SHA256:/      { sub(/^SHA256: /,"");   SHA256=$0; }
/^Depends:/     { sub(/^Depends: /,"");     PKGDEP=fixdepends($0) "," PKGDEP; }
/^Pre-Depends:/ { sub(/^Pre-Depends: /,""); PKGDEP=fixdepends($0) "," PKGDEP; }
/^Description:/ { sub(/^Description: /,""); PKGINFO=substr($0,1,200); }
/^$/            { print PKG "_" PKGVER "|" PKG "|" PKGVER "||" PKGSECTION "|" PKGSIZE "|" PKGPATH "|" PKGFILE  "|" PKGDEP "|" PKGINFO "|" PKGOS "|" PKGOSVER "|"  PKGMD5 "|" SHA1 "|" SHA256 "|" PKGPRIO "|" REPOFNM  ;
   
                  PKG=""; PKGVER=""; PKGSECTION=""; PKGSIZE=""; PKGFILE=""; PKGPATH=""; PKGDEP=""; PKGINFO=""; PKGPRIO="";  PKGMD5=""; SHA1=""; SHA256=""; PKGDEP="";  }
' > ~/awk_dep_test.out

https://pastebin.com/FsG3y7Ca

version with debuging print statments:
https://pastebin.com/bHpDuXH3

Sample input file
Code:

Package: ntfs-3g
Version: 1:2016.2.22AR.1+dfsg-1+deb9u1
Installed-Size: 1433
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Architecture: i386
Depends: libc6 (>= 2.17), libgcrypt20 (>= 1.7.0), libgnutls30 (>= 3.5.0), libgpg-error0 (>= 1.14), libntfs-3g871 (= 1:2016.2.22AR.1+dfsg-1+deb9u1)
Pre-Depends: fuse
Homepage: http://www.tuxera.com/community/ntfs-3g-advanced/
Description: read/write NTFS driver for FUSE
Description-md5: d38b96403238266210560bdd2ee0d28c
Tag: admin::filesystem, hardware::storage, implemented-in::c,
 interface::commandline, interface::shell, role::program, scope::utility
Section: otherosfs
Priority: optional
Filename: pool/DEBIAN/main/n/ntfs-3g/ntfs-3g_2016.2.22AR.1+dfsg-1+deb9u1_i386.deb
Size: 427186
MD5sum: 8060e6cd9106d6b366473abe57a5af38
SHA256: 662a28b55f4f770cc815a258a15629fb47684da59d456dc4fa3fa4f136c2a339

https://pastebin.com/nx5GcD6h

If one doesn't get any output add an extra line or two to the end of the input file. This is required because the awk code only prints the output when there is a blank line.

Here is the output:

Code:

ntfs-3g_1:2016.2.22AR.1+dfsg-1+deb9u1|ntfs-3g|1:2016.2.22AR.1+dfsg-1+deb9u1||otherosfs|1433|pool/DEBIAN/main/n/ntfs-3g|ntfs-3g_2016.2.22AR.1+dfsg-1+deb9u1_i386.deb|+fuse,+libc6&ge2.17,+libgcrypt20&ge1.7.0,+libgnutls30&ge3.5.0,+libgpg-error0&ge1.14,+libntfs-3g871&eq1:2016.2.22AR.1+dfsg-1+deb9u1,|read/write NTFS driver for FUSE|devuan|ascii|8060e6cd9106d6b366473abe57a5af38||662a28b55f4f770cc815a258a15629fb47684da59d456dc4fa3fa4f136c2a339|optional|/var/packages/repo/Packages-devuan-ascii-main


Notes on pastebin

Pastebin does not use unix style line endings. Therefore either copy and paste the code into a text editor or ide, or alternatively use the dos2unix utility if you download the file. You can also convert line endings in some IDEs (e.g. geany) and text editors.

Notes on dependencies

I believe that since I'm using arrays in my awk code that gawk is required. However, I think that busybox might actually have gawk as part of it. Therefore, it is possible that the busybox version of gawk might work (need to verify).

Credits and thankyou's

Thank you to jamesbond, because the awk code I started with was the code that he created in woof next. Thank you to technosaurus for the link to the regular expression tester (see post):

https://regex101.com/r/8hP5lS/1

Without this tester it would have taken me considerably longer to get this code to work, assuming that I got it to work at all!

Finally, thankyou to sc0ttman for the "pkg" project, which is a very powerful package manager.

Previous posts on awk
- AWK: match($2,/^(.*[^:digit:])([:digit:]*$|$)/,pkg_split)
- AWK Based Version Comparison

Other links that will be useful to me for future development
- busybox getopt ignores 1st positional paramater

Related development Topics
- Using gpg/pgp signatures in Package Managers
- Validating Package Archives and Metadata
- See also post in Puli 32/64 bit thread, and post in tiny_puduan thread.

_________________
Find me on minds and on pearltrees.

Last edited by s243a on Tue 26 Nov 2019, 16:27; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
s243a

Joined: 02 Sep 2014
Posts: 2478

PostPosted: Tue 26 Nov 2019, 16:21    Post subject:  

I pushed these changes to github (commit 8e44442):

/s243a/.../usr/sbin/ppa2pup#L347

I also added a hack to help pkg_get() find the downloaded file (see: Issue #73 note_250358773) but this requires further testing.

Code:

ln -s "${PKG_FILENAME}" "$PKG_FNAME_GUESS" #TODO use relative symbolic links + symbolic link cleanup.

/s243a/.../usr/sbin/pkg-classic#L4619

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
oui

Joined: 20 May 2005
Posts: 3529
Location: near Woof (Germany) :-) - 3 PC's: DELL SX280 750 MB Pentium4, Acer emachines 2 GB AMD64. DELL XPS15

PostPosted: Tue 26 Nov 2019, 16:57    Post subject:  

great, thank you for your development efforts (auxiliary question: the opposite one? some "convert-pet2deb"?)
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 2478

PostPosted: Tue 26 Nov 2019, 17:06    Post subject:  

oui wrote:
great, thank you for your development efforts (auxiliary question: the opposite one? some "convert-pet2deb"?)


I just googled pet2deb and got this:

saintless wrote:
Included in the latest DebianDog from 16.06.2014 - pet2sfs and pet2deb converting scripts.
http://smokey01.com/saintless/DebianDog/Packages/Included/pet2sfs-pet2deb-scripts.tar.gz
All File managers have right click option for converting pet file or directory with pet files.
For example right click on pet package or directory in Xfe -> Scripts -> and choose:
pet2sfs-convert - convert single pet to sfs
pet2sfs-dir-convert - convert directory with pet to sfs
pet2deb-convert - convert single pet to deb
pet2deb-dir-convert - convert directory with pet to deb

Or from terminal:
Code:
pet2sfs <package>.pet

Load the sfs to test it and if it works you can make deb package for installing/uninstalling:
Code:
pet2deb <package>.pet


Both scripts keep the pet package untouched and auto-remove the working folders. The new deb or sfs file is in the same directory where the pet is. The process is automated without actions from the user.

pet2sfs does not change the pet package structure. It just removes pet.specs file and makes sfs from the working folder.

pet2deb (modded version of redeb + part of make-deb-package from RSH) changes the structure by moving executable file in /opt/bin and libs in /opt/lib. Then adds DEBIAN folder with control and postinst script to update-menus.

http://www.murga-linux.com/puppy/viewtopic.php?p=783785#783785

_________________
Find me on minds and on pearltrees.
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 1 of 1 [4 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0543s ][ Queries: 12 (0.0095s) ][ GZIP on ]