Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 19 Dec 2014, 10:30
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
speeding up scripts
Post new topic   Reply to topic View previous topic :: View next topic
Page 2 of 3 [43 Posts]   Goto page: Previous 1, 2, 3 Next
Author Message
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Sun 14 Aug 2011, 19:14    Post subject:  

I just wanted to mention that almost all of the external binary advice goes out the window if you use a busybox shell (ash/hush) with the busybox_prefer_applets option enabled. No major x86 distro, with the exception of maybe alpine linux, enables it by default though.

here is a template that I use to parse desktop files
Code:
#!/bin/ash
#Copywrong 2011 Brad Conroy - released to the public domain
#Parse through the .desktop files, generate use{ful,less} output
#Though they have Var=value structure, we cannot source them directly (damn) because:
#1. many Vars have illegal characters ... but not the ones we actually want (for now - see TODO)
#2. many values have spaces
#Solution
#1. grep for only the fields that we want to eliminate problem #1
#2.(a) use sed to put quotes after the = and at the end of the line to accomodate spaces
#2.(b) ayttm (possibly others) has a non-standard set of quotes so add a sed for that (could tr -s '"',but sed is already called)
#But wait we didn't _actually_ modify the file, so we can't source it - what a waste of time
#That's ok, we can just use eval on a variable (or in this case a return) to do the same thing
#
#Ok now we have the values, but what to do with them?  We will just output them to an easily awkable file
#but writing to a file is slow, so store it in a Variable in-loop and write once out-of-loop
#... at this point you could instead add a single case statement to generate menu entries or anything useful
#TODO add parameters to the sed/grep parts to localize if available (trivial, but would complicate the demo)
for x in /usr/share/applications/* ; do
    eval `grep -E ^Name=\|^Categories=\|^Comment=\|^Icon= $x |sed "s/=/=\"/g ; s/$/\"/g ; s/\"\"/\"/g"`
    OUTPUT=${OUTPUT}${Name:-UNDEFINED}\|${Categories:-UNDEFINED}\|${Comment:-UNDEFINED}\|${Icon:-UNDEFINED}"\n"
done
echo -e $OUTPUT

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Mon 15 Aug 2011, 06:16    Post subject:  

Code:
#this time with fgrep, it needs newline separators and doesn't support "^" (beginning of line)
#This pulls in GenericName and other problem children ... just for demo, then back to grep -E or egrep
#... thus the 4th sed (a blank entry becomes ="$ so fix it)
#This time we will thread the parsing of each file - it is ~10x faster but borks sorting by *.desktop name
#perhaps we want to sort by app name or category ... or, if you are using bash, store them in an mxn array

String="Name=
Categories=
Comment=
Icon=
Exec="

parse(){ #by pulling this out of the main loop we can fork it with the & and run many simultaneously
eval $(fgrep "$String" ${1} |sed 's/=/=\"/g ; s/$/\"/g ; s/\"\"/\"/g ; s/=\"$/=\"\"/')
echo ${Name:-UNDEFINED}\|${Categories:-UNDEFINED}\|${Comment:-UNDEFINED}\|${Icon:-UNDEFINED}\|${Exec:-UNDEFINED}
}

printf "" > /tmp/file #start with a clean file
for x in /usr/share/applications/* ; do
 parse "${x}" 2>/dev/null >>/tmp/file &
done

#hmm the recursing of the files is done, but what about all of the threads?
#lets try sorting the file and see?
#sort file > ${1:-sortedfile}
#nope missing entries, need to wait ... but how long
#don't just make a sufficiently long sleep for _your_ box
#we will just wait for sed to complete

while (pidof sed >/dev/null) do
printf . #as long as sed is running print dots every 10 milliseconds
sleep 0.01
done
sort /tmp/file > ${1:-sortedfile}

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2290

PostPosted: Mon 15 Aug 2011, 06:41    Post subject:  

"grep -F by default". Yeah that's okay. Of course it's an extra process since it's a wrapper. On my older Slack, fgrep is a *link* to grep. But ony my own kiss-linux I install the real binary of fgrep. It's a smaller bin so has less latency.

Techno's bit of code makes a good example. I compared it to a pure-shell solution which gives the same output:

Code:
#/bin/bin/bsh
# this runs fine on 'real' ash and should work with bb ash
#/usr/bin/bsh
# It will also run using 'bsh' the 'real' Bourne shell
# Quite obvioulsy would run with any bash, as well.


#Copywrong 2011 Brad Conroy - released to the public domain
#Parse through the .desktop files, generate use{ful,less} output
#Though they have Var=value structure, we cannot source them directly (damn) because:
#1. many Vars have illegal characters ... but not the ones we actually want (for now - see TODO)
#2. many values have spaces
#Solution
#1. grep for only the fields that we want to eliminate problem #1
#2.(a) use sed to put quotes after the = and at the end of the line to accomodate spaces
#2.(b) ayttm (possibly others) has a non-standard set of quotes so add a sed for that (could tr -s '"',but sed is already called)
#But wait we didn't _actually_ modify the file, so we can't source it - what a waste of time
#That's ok, we can just use eval on a variable (or in this case a return) to do the same thing
#
#Ok now we have the values, but what to do with them?  We will just output them to an easily awkable file
#but writing to a file is slow, so store it in a Variable in-loop and write once out-of-loop
#... at this point you could instead add a single case statement to generate menu entries or anything useful
#TODO add parameters to the sed/grep parts to localize if available (trivial, but would complicate the demo)
function techno() {
for x in /usr/share/applications/* ; do
    eval `grep -E ^Name=\|^Categories=\|^Comment=\|^Icon= $x |sed "s/=/=\"/g ; s/$/\"/g ; s/\"\"/\"/g"`
    OUTPUT=${OUTPUT}${Name:-UNDEFINED}\|${Categories:-UNDEFINED}\|${Comment:-UNDEFINED}\|${Icon:-UNDEFINED}"\n"
done
echo -e $OUTPUT
}
#time techno
# On my 700mHz P-III the above runs in
#real    0m0.882s
#user    0m0.380s
#sys     0m0.430s
# on a dir with 56 items
# Hmm, that echo -e may not be enirely portable, but you could use printf or /bin/echo instead

#Copyright Gilbert Ashley <amigo@ibiblio.org>
amigo() {
> test.file
for DESKTOP_FILE in /usr/share/applications/* ; do
#for DESKTOP_FILE in /usr/share/applications/Editra.desktop ; do
   OUT=
   while read LINE ; do
      case $LINE in
         Name=*) NAME="${LINE#*=}"'|'  ;;
         Comment=*) OUT=$OUT"${LINE#*=}"'|'  ;;
         Icon=*) ICON="${LINE#*=}"'|'  ;;
         #Terminal=*)
         #Type=*)
         Categories=*) CATS="${LINE#*=}"'|'  ;;
         Exec=*) EXEC="${LINE#*=}"'|'  ;;
         #Comment=*) COMM="${LINE#*=}"  ;;
      esac
   done < $DESKTOP_FILE
   echo $NAME$ICON$CATS$EXEC
   # To test the extract function below, use the following line instead of above
   # echo $NAME$ICON$CATS$EXEC >> test.file
done
}

time amigo
# On my 700mHz P-III the above runs in
#real    0m0.160s
#user    0m0.130s
#sys     0m0.010s
# on a dir with 56 items

# And about 'awkable', there again we can do that with pure shell:
extract() {
while read LINE ; do
   # skip the line if NULL
   case $LINE in '') continue ;; esac
   
   # more conventional approaches awk:
   #echo $LINE | awk -F '|' '{ print $4 }'
   #real    0m0.684s
   #user    0m0.290s
   #sys     0m0.330s
   
   # and with cut:
   #echo "$LINE" |cut -f4 -d'|'
   #real    0m0.613s
   #user    0m0.200s
   #sys     0m0.350s

   ( IFS='|' ; set ${LINE} ; echo $4 )
   # Using ash:
   #real    0m0.298s
   #user    0m0.070s
   #sys     0m0.170s
   # Using bash:
   #real    0m0.296s
   #user    0m0.080s
   #sys     0m0.120s
   # Using bsh:
   #real    0m0.285s
   #user    0m0.060s
   #sys     0m0.110s

done < test.file
}

time extract



Should run even faster using bb ash since it's all a single process. I'd point out, that even though you are using bb grep and sed(right?), you are still running a separate process for each one.
Back to top
View user's profile Send private message 
PANZERKOPF

Joined: 16 Dec 2009
Posts: 280
Location: Earth

PostPosted: Mon 15 Aug 2011, 09:13    Post subject:  

technosaurus wrote:
busybox_prefer_applets option enabled.

That is great option but seems we must mount /proc before using it.
After making this in sysinit:
/bin/busybox mount -t proc proc /proc
We can call any builtin application directly.
Is that right?

_________________
SUUM CUIQUE.
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Mon 15 Aug 2011, 15:42    Post subject:  

PANZERKOPF wrote:
technosaurus wrote:
busybox_prefer_applets option enabled.

That is great option but seems we must mount /proc before using it.
After making this in sysinit:
/bin/busybox mount -t proc proc /proc
We can call any builtin application directly.
Is that right?
I don't know about _any_, but most (some _may_ need other stuff like sys/dev or specific files/nodes) however, if you have busybox ash or hush as /bin/sh and a shell-compliant-script relies on standard utils, it will fail ... you'd need to mod it from util <args> to /path/to/util <args> for those instances ... which is why no major distro is doing it
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Mon 15 Aug 2011, 22:17    Post subject:  

Thanks to amigo for the read line trick.
I used it to throw together a jwm_menu_create script that completes in 1/15th the time of fixmenus using only shell commands (and one echo -e, because some of Puppy's busyboxes don't have printf)

usage:
jwm_menu_create > ${HOME}/.jwmrc && jwm -restart #by default it will just go to stdout

Todo:
1. What the heck is up the default*.desktop file stubs - do I need a work around or are they misplaced/malformed (they are missing most entries)
2. Localization - just needs another case statement, but few .desktop files support it anyways ... and my Locale is already default Smile

Edit It is now in this thread
http://www.murga-linux.com/puppy/viewtopic.php?t=70804

_________________
Web Programming - Pet Packaging 100 & 101

Last edited by technosaurus on Tue 16 Aug 2011, 05:00; edited 1 time in total
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2290

PostPosted: Tue 16 Aug 2011, 02:51    Post subject:  

"1/15th the time of fixmenus" -Well, I guess it was worth the effort. One time I re-wrote something and it ran in 3% of the former time -even though neither routine used any externals.
Back to top
View user's profile Send private message 
jamesbond

Joined: 26 Feb 2007
Posts: 2232
Location: The Blue Marble

PostPosted: Tue 16 Aug 2011, 04:32    Post subject:  

Thanks, this is very educational. I don't use the jwm so the script isn't much help for me, but the idea behind it is very helpful for future projects.
_________________
Fatdog64, Slacko and Puppeee user. Puppy user since 2.13.
Contributed Fatdog64 packages thread
Back to top
View user's profile Send private message 
sc0ttman


Joined: 16 Sep 2009
Posts: 2386
Location: UK

PostPosted: Tue 16 Aug 2011, 09:36    Post subject:  

jamesbond wrote:
Thanks, this is very educational.

Once again I started a thread I'm too thick to follow Laughing This thread went over my head about 2 pages ago.. I'm gonna read, re-read, and re-read again until it makes sense! But I'm still loving the results! Smile

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
thunor


Joined: 14 Oct 2010
Posts: 350
Location: Minas Tirith, in the Pelennor Fields fighting the Easterlings

PostPosted: Tue 16 Aug 2011, 17:16    Post subject: Re: speeding up scripts
Subject description: waht I learned, what I wanna learn
 

sc0ttman wrote:
1. Is it possible to execute a command defined in the main script, from within a GTK-Dialog button? If so, that would be great, but at the moment, anything like this in the GTKDialog GUI
Code:

<button>
<action>my_function_name</action>
</button>
simply returns (in the terminal)
Code:
my_function_name: command not found

Does <action>exec $SHELL -c 'my_function_name'</action> solve it? Somebody using Ubuntu+dash reported a similar problem and this was his fix.

Regards,
Thunor
Back to top
View user's profile Send private message Visit poster's website 
sc0ttman


Joined: 16 Sep 2009
Posts: 2386
Location: UK

PostPosted: Wed 17 Aug 2011, 08:30    Post subject: Re: speeding up scripts
Subject description: waht I learned, what I wanna learn
 

thunor wrote:
sc0ttman wrote:
1. Is it possible to execute a command defined in the main script, from within a GTK-Dialog button? If so, that would be great, but at the moment, anything like this in the GTKDialog GUI
Code:

<button>
<action>my_function_name</action>
</button>
simply returns (in the terminal)
Code:
my_function_name: command not found

Does <action>exec $SHELL -c 'my_function_name'</action> solve it? Somebody using Ubuntu+dash reported a similar problem and this was his fix.

Regards,
Thunor

I'll have a look, thanks for the tip Smile

_________________
Akita Linux, VLC-GTK, Pup Search, Pup File Search
Back to top
View user's profile Send private message 
frafa


Joined: 04 Aug 2011
Posts: 10
Location: MONTPELIER

PostPosted: Wed 17 Aug 2011, 09:10    Post subject:  

Hello,
you can simplify in:
Code:
<action>bash -c 'my_function_name'</action>

it's because of sub shell, in Ubuntu is dash
and dash does not support "export-f"

do the same for <input>
if <input> is a function
Back to top
View user's profile Send private message Visit poster's website 
big_bass

Joined: 13 Aug 2007
Posts: 1747

PostPosted: Thu 18 Aug 2011, 16:26    Post subject:  

speeding up scripts using arrays


when you have correctly formatted files there is another way to pull out values


*so if you start out creating a correctly formatted file
getting info is easy


here is an example using a .desktop
we all know that desktops are not formatted correctly or is there an official
line by line template that is used by all linux distros

but we could make a template Very Happy

then read it with this
using shell and arrays
--------------------------------------------------------------------------------------------

Code:


# advanced arrary use from info in advanced bash scripting modified by Joe Arose
# for using correctly formatted templates

# this separates the strings using a pipe symbol this allows for spaces in strings to be read
# to later be converted back to spaces
# if not done this way every space is a new arrary value causing undesired word splitting
# this fixes many problems with getting just one line of information

 
#this is beautiful it replaces the pipe for a space
#echo ${desktop_array[4]//|/ }
 
#how that reads is echo array 4 and  //substitue all , pipe symbols, with a space
# it looks a bit like sed but using only the shell 


# set this to what you want
fileplace=/usr/share/applications/
filename=Axel-download-accelerator.desktop



desktop_array=( `cat "$fileplace$filename" | tr ' ' '|'`)


echo ${desktop_array[@]//|/ }

echo ${desktop_array[0]//|/ }

echo ${desktop_array[1]//|/ }

echo ${desktop_array[2]//|/ }

echo ${desktop_array[3]//|/ }

echo ${desktop_array[4]//|/ }

echo ${desktop_array[5]//|/ }

echo ${desktop_array[6]//|/ }

echo ${desktop_array[7]//|/ }

echo ${desktop_array[8]//|/ }

echo ${desktop_array[9]//|/ }

echo ${desktop_array[10]//|/ }




sample output

# echo ${desktop_array[0]//|/ }
[Desktop Entry]
#
# echo ${desktop_array[1]//|/ }
Encoding=UTF-8
#
# echo ${desktop_array[2]//|/ }
Name=Axel download accelerator
#
# echo ${desktop_array[3]//|/ }
Icon=mini-ftp.xpm
#
# echo ${desktop_array[4]//|/ }
Comment=Axel download accelerator
#
# echo ${desktop_array[5]//|/ }
Exec=puppydownload
#
# echo ${desktop_array[6]//|/ }
Terminal=false
#
# echo ${desktop_array[7]//|/ }
Type=Application
#
# echo ${desktop_array[8]//|/ }
Categories=X-Internet
#
# echo ${desktop_array[9]//|/ }
GenericName=Axel download accelerator
#

_________________
debian wheezy ,linux mint, slackware I use them all and they all have good points
Mint would be best for general users though
Back to top
View user's profile Send private message 
big_bass

Joined: 13 Aug 2007
Posts: 1747

PostPosted: Fri 19 Aug 2011, 19:18    Post subject:  

I modified amigo's code to make desktops now write to and read from an array

*thanks for that snippet Gilbert it was really clever using the case for a pre filter grep and very fast
Code:

> test.file
for DESKTOP_FILE in /usr/share/applications/* ; do
#for DESKTOP_FILE in /usr/share/applications/Editra.desktop ; do
    while read LINE ; do
      case $LINE in
         Name=*) NAME="${LINE[@]}"'|'   ;;
         Icon=*) ICON="${LINE[@]}"'|'   ;;
         #Terminal=*)
         #Type=*)
         Categories=*) CATS="${LINE[@]}"'|'   ;;
         Exec=*) EXEC="${LINE[@]}"'|'   ;;
         Comment=*) COMM="${LINE[@]}"'|'   ;;
      esac
   done < $DESKTOP_FILE
   echo $NAME$ICON$CATS$EXEC
   # To test the extract function below, use the following line instead of above
    echo '[Desktop_Entry]|'$NAME$ICON$CATS$EXEC$COMM | tr ' ' '_'>> test.file
done


desktop_array2=( `cat "$HOME/test.file"`)


# read line by line of the formatted array
# one problem was the Name= and the comment didnt get the spaces converted into
# the pipe symbol because the pipe was after the string with spaces
# but this gets fixed with underscores in the last echo using tr ' ' '_'


# how to read line by line from a  correctly  forrmatted  array now
# just a sample of 5 diferent desktops read from the array desktop_array2

echo ${desktop_array2[0]//|/  } | tr ' ' '\n' | tr '_' ' '>testing0.txt
echo ${desktop_array2[1]//|/  } | tr ' ' '\n' | tr '_' ' '>testing1.txt
echo ${desktop_array2[2]//|/  } | tr ' ' '\n' | tr '_' ' '>testing2.txt
echo ${desktop_array2[3]//|/  } | tr ' ' '\n' | tr '_' ' '>testing3.txt
echo ${desktop_array2[4]//|/  } | tr ' ' '\n' | tr '_' ' '>testing4.txt






original code from amigo below
Code:
> test.file
for DESKTOP_FILE in /usr/share/applications/* ; do
#for DESKTOP_FILE in /usr/share/applications/Editra.desktop ; do
   OUT=
   while read LINE ; do
      case $LINE in
         Name=*) NAME="${LINE#*=}"'|'  ;;
         Comment=*) OUT=$OUT"${LINE#*=}"'|'  ;;
         Icon=*) ICON="${LINE#*=}"'|'  ;;
         #Terminal=*)
         #Type=*)
         Categories=*) CATS="${LINE#*=}"'|'  ;;
         Exec=*) EXEC="${LINE#*=}"'|'  ;;
         #Comment=*) COMM="${LINE#*=}"  ;;
      esac
   done < $DESKTOP_FILE
   echo $NAME$ICON$CATS$EXEC
   # To test the extract function below, use the following line instead of above
   # echo $NAME$ICON$CATS$EXEC >> test.file
done

_________________
debian wheezy ,linux mint, slackware I use them all and they all have good points
Mint would be best for general users though

Last edited by big_bass on Fri 19 Aug 2011, 19:39; edited 3 times in total
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4424

PostPosted: Fri 19 Aug 2011, 19:32    Post subject:  

If you use arrays, make sure to change the shabang to /bin/bash as they are bashisms.
_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 2 of 3 [43 Posts]   Goto page: Previous 1, 2, 3 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1346s ][ Queries: 13 (0.0067s) ][ GZIP on ]