Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Wed 23 Apr 2014, 13:54
All times are UTC - 4
 Forum index » Off-Topic Area » Programming
speeding up scripts
Post new topic   Reply to topic View previous topic :: View next topic
Page 3 of 3 [43 Posts]   Goto page: Previous 1, 2, 3
Author Message
big_bass

Joined: 13 Aug 2007
Posts: 1747

PostPosted: Mon 22 Aug 2011, 15:47    Post subject:  

I only use bash
* but if you are using another shell this runs only in bash*
you can take advantage of arrays using bash

(I credited amigo in a prior thread for the code base and pasted the original code a few posts ago )* I will include a diff patch for clarity

I put a safety in there all files get generated in /tmp/desktop
so if you are happy with the results you could manually copy over the
old desktops (thats your call) the user woud have to repackage all the packages with the new desktops to do the job right

I replaced the underscore with the "+" because some files may use underscore in names

Code:
#!/bin/bash

make a template of all the desktops and lets you view them in /tmp/desktops
so none of your original desktops get overwritten

builds an array for speeding up any scripts that search info from the desktop

now all your desktops will have a format and an organized  template
the first line is [Desktop Entry],Name,Icon,Categories,Exec,Comment

and thats what you will expect to see when you search the desktops for info 


#[Desktop Entry]
#Name=AbiWord
#Icon=/usr/share/icons/abiword_48.png
#Categories=Application;Office;WordProcessor;GNOME;GTK;X-Red-Hat-Base;
#Exec=abiword
#Comment=Compose, edit, and view documents





# make a template  of the desktops regenerate all desktops  to the new simple template
# removes poorly formatted desktops and creates a standard  which allows later  for easy reading of strings
# into an array to speed up scripts since the newly generated desktops maintain a standard format
# less commands are needed to filter data for output this is where all the time is wasted
# having to parse poorly formatted files from the start if you have organized files
# everything is fast and easy to  parse the code

:>arraytest.txt
mkdir -p /tmp/desktops

for DESKTOP_FILE in /usr/share/applications/* ; do
#for DESKTOP_FILE in /usr/share/applications/Editra.desktop ; do
    while read LINE ; do
      case $LINE in
         Name=*) NAME="${LINE[@]}"'|'   ;;
         Icon=*) ICON="${LINE[@]}"'|'   ;;
         #Terminal=*)
         #Type=*)
         Categories=*) CATS="${LINE[@]}"'|'   ;;
         Exec=*) EXEC="${LINE[@]}"'|'   ;;
         Comment=*) COMM="${LINE[@]}"'|'   ;;
      esac     
   done < $DESKTOP_FILE
   echo $NAME$ICON$CATS$EXEC
   # To test the extract function below, use the following line instead of above
   # fixes spaces in the string names by replacing them with a "+" making a correctly formatted array
    echo '[Desktop+Entry]|'$NAME$ICON$CATS$EXEC$COMM | tr ' ' '+'  >>arraytest.txt
    #uncomment if you want to generate all new desktops in /temp/desktops
    echo '[Desktop+Entry]|'$NAME$ICON$CATS$EXEC$COMM | tr ' ' '+' | tr '| ' ' ' | tr ' ' '\n' | tr '+' ' '>/tmp/desktops/`basename $DESKTOP_FILE`
done
amigo-orig.patch.gz
Description  this is only to show the differences between amigos original work and my modifications
gz

 Download 
Filename  amigo-orig.patch.gz 
Filesize  737 Bytes 
Downloaded  140 Time(s) 

_________________
debian wheezy ,linux mint, slackware I use them all and they all have good points
Mint would be best for general users though
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4134

PostPosted: Wed 24 Aug 2011, 00:45    Post subject:  

here is an example I cooked up that includes recursion, string manipulation, and integer math

Code:
#!/bin/ash
A=$(($1/16))
HEX=0123456789ABCDEF
[ $A -gt 15 ] && dec2hex ${A} || printf ${HEX:$A:1}
printf ${HEX:$(($1%16)):1}


A=$(($1/16))
$1 is the input, this stores the "div" of the input by 16 (div is integer only without a remainder so 15 div 16 is 0 but 17 div 16 is 1)

HEX=0123456789ABCDEF
this shows how strings are really just an array of characters

[ $A -gt 15 ] && dec2hex ${A} || printf ${HEX:$A:1}
this is the recursive part, if the "div" is greater than 16, then we haven't gone enough hex place values, so call ourself with the div to shift back one ... note that nothing further happens until the last place value is reached (div is < 16) and the all recursive calls to dec2hex return... this last one only will print its div (in HEX format) the value after the first ":" is the starting point of the substring, and the value after the second ":" is the length of the substring

printf ${HEX:$(($1%16)):1}
similar to above printf statement except that it prints the "mod" (the remainder of input div 16) ... notice that all recursive calls will execute this code

now just to show the value of using functions instead of external scripts

try it like this:

Code:
#!/bin/ash

dec2hex(){
A=$(($1/16))
HEX=0123456789ABCDEF
[ $A -gt 15 ] && dec2hex ${A} || printf ${HEX:$A:1}
printf ${HEX:$(($1%16)):1}
}

dec2hex $1


[list]# time dec2hex 999999999999999999
DE0B6B3A763FFFF
real 0m0.034s
user 0m0.032s
sys 0m0.024s
# time dec2hex 999999999999999999

and if you want a generalized format for any base
Code:

#!/bin/ash

#note that base64 is traditionally A...Za...z0...9+/ (yeah wtf)
dec2baseN(){
A=$(($1/$2))
STRING=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/
[ $A -ge $2 ] && dec2baseN ${A} $2 || printf ${STRING:$A:1}
printf ${STRING:$(($1%$2)):1}
}

dec2baseN $1 $2

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
disciple

Joined: 20 May 2006
Posts: 6378
Location: Auckland, New Zealand

PostPosted: Tue 30 Aug 2011, 20:49    Post subject:  

Does anyone know: if you source another file, are functions from it run in a new subshell? I don't see why they would be... but I don't know.
_________________
DEATH TO SPREADSHEETS
- - -
Classic Puppy quotes
- - -
Beware the demented serfers!
Back to top
View user's profile Send private message 
technosaurus


Joined: 18 May 2008
Posts: 4134

PostPosted: Tue 30 Aug 2011, 21:19    Post subject:  

No, but it does take considerably longer, due to the extra file read.

Things that should be sourced include (not limited to...just examples
Localization (b/c you are only source 1 of X)
Configuration files (can be used/modified by other programs or the user)
A single file that contains all of your needed functions.

Things that usually shouldn't be sourced
Lots of files with a single or a few functions (each read adds time)
A self generated file (you can normally use a variable)

_________________
Web Programming - Pet Packaging 100 & 101
Back to top
View user's profile Send private message 
disciple

Joined: 20 May 2006
Posts: 6378
Location: Auckland, New Zealand

PostPosted: Tue 30 Aug 2011, 21:43    Post subject:  

Quote:
No

Ah, thanks, I confirmed that by testing, too Smile

I don't know if anybody here would find it useful, but I see the Arch people maintain "A library providing UI functions for shell scripts"
https://github.com/Dieterbe/libui-sh#readme

_________________
DEATH TO SPREADSHEETS
- - -
Classic Puppy quotes
- - -
Beware the demented serfers!
Back to top
View user's profile Send private message 
amigo

Joined: 02 Apr 2007
Posts: 2171

PostPosted: Wed 31 Aug 2011, 04:28    Post subject:  

When you source a file, it is just as if you 'pasted' the code from it into wherever you sourced in your calling script.

I have one very large shell project -well over 10,000 lines of code. It used all be in one file, but when it got over 3,000 lines I began splitting it up. I do like using functions greatly and name the files after the main function they contain. But, I wouldn't do this for a single small function. In my large project, some of the files containing functions are sourced selectively, according to whether the functions are needed or not.

About the time needed for reading the file in -it takes time each time you check a file attribute using 'test' or '[' and we never shy from that right? And, reading in the contents of the file won't take much longer. And it usually takes much less time than starting some external program -the startup latency from lodaing that into the cache and linking usually takes longer than simply opening a file for read.

Thanks for mentioning 'libui-sh'. I don't think I had seent had one before. There are a couple of others, also, which include a standard set of dialog boxes that will work with several different tools -dialog, Xdialog, zenity, gdialog or even just plain shell. I've found that most of them are too big and overreaching. The ideal setup for creating dialogs would cover a few standard screens like menu-selection, input-box, info-box, yes-no, etc.

I've put lots of time into learning to cerate faster scripts by focusing on replacing often-used standard tools like basename, dirname, cut, cat, etc., with small blocks of bash code.

basename?
${VAR##*/}
dirname?
${VAR%/*}
cat?
while something ; do

done < file-you-wanted-to-cat
Back to top
View user's profile Send private message 
disciple

Joined: 20 May 2006
Posts: 6378
Location: Auckland, New Zealand

PostPosted: Tue 06 Sep 2011, 08:18    Post subject:  

Does anyone know if there is a difference between something like `sed -i ... somefile` and `sed ... < somefile`?
I don't think there is, but then why are there both options? Is it because the < isn't available in certain shells or something?

_________________
DEATH TO SPREADSHEETS
- - -
Classic Puppy quotes
- - -
Beware the demented serfers!
Back to top
View user's profile Send private message 
big_bass

Joined: 13 Aug 2007
Posts: 1747

PostPosted: Tue 06 Sep 2011, 09:16    Post subject:  

the sed -i

makes the changes in the "original file" permanent

and the other command without the -i just lets you see what would happen
only the terminal shows the results the original file is unchanged


sed is a great tool but... it looks like taking modem noise and then converting it
into a human readable language

sed <modem_noise # is used to generate the command options
sed1line.txt.gz
Description 
gz

 Download 
Filename  sed1line.txt.gz 
Filesize  6.87 KB 
Downloaded  124 Time(s) 

_________________
debian wheezy ,linux mint, slackware I use them all and they all have good points
Mint would be best for general users though
Back to top
View user's profile Send private message 
disciple

Joined: 20 May 2006
Posts: 6378
Location: Auckland, New Zealand

PostPosted: Tue 06 Sep 2011, 09:57    Post subject:  

True! I guess I should be in bed. The reason I wasn't seeing any difference is because the line in the script I was working on wasn't actually necessary for the data I was testing it with Rolling Eyes
Thanks.

_________________
DEATH TO SPREADSHEETS
- - -
Classic Puppy quotes
- - -
Beware the demented serfers!
Back to top
View user's profile Send private message 
thunor


Joined: 14 Oct 2010
Posts: 350
Location: Minas Tirith, in the Pelennor Fields fighting the Easterlings

PostPosted: Sun 04 Dec 2011, 10:52    Post subject:  

I've got some fast portable shell scripting tips that I'd like to note.

Code:
## With bash you can read the first line from a file speedily like this:

echo `<file`

## This you might think is the ash/dash solution:

cat file

## But I've actually found this to be the fastest for ash, bash and dash:

read -r input < file
echo $input

I'm off out in a minute so I'll add some more later.

Regards,
Thunor

Last edited by thunor on Sun 04 Dec 2011, 19:16; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
jpeps

Joined: 31 May 2008
Posts: 3219

PostPosted: Sun 04 Dec 2011, 12:23    Post subject:  

thunor wrote:
I've got some fast portable shell scripting tips that I'd like to note.

Code:
## With bash you can read a file speedily like this:

echo `<file`

## This you might think is the ash/dash solution:

cat file

## But I've actually found this to be the fastest for ash, bash and dash:

read -r input < file
echo $input


That prints only the first line ??

echo `<file` doesn't preserve the line feed. "cat file" is faster (on my computer)
Back to top
View user's profile Send private message 
thunor


Joined: 14 Oct 2010
Posts: 350
Location: Minas Tirith, in the Pelennor Fields fighting the Easterlings

PostPosted: Sun 04 Dec 2011, 19:15    Post subject:  

jpeps wrote:
That prints only the first line ??

echo `<file` doesn't preserve the line feed. "cat file" is faster (on my computer)

Yeah, I should've mentioned that I was only interested in the first line such as when maintaining global variables as files Smile

Getting the directory name from a path:

Code:
## This is what you might be tempted to do:

fullpath="/some/path/to some file.txt"
path="`dirname "$fullpath"`"

## But this is portable although you'll need to check that it returns at least root:

fullpath="/some/path/to some file.txt"
path="${fullpath%/*}"
if [ -z "$path" ]; then path="/"; fi

Regards,
Thunor

Last edited by thunor on Sun 04 Dec 2011, 19:51; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website 
thunor


Joined: 14 Oct 2010
Posts: 350
Location: Minas Tirith, in the Pelennor Fields fighting the Easterlings

PostPosted: Sun 04 Dec 2011, 19:47    Post subject:  

Checking the first character of a string:

Code:
## You can get the first character using bash like this:

char="${path:0:1}"

## Otherwise you can use this (is there a better way?):

char="`echo $path | cut -c 1`"

## But if you know what you're looking for e.g. making sure that a string has an initial forward slash then you can do this:

case "$path" in
   /*) true ;;
   *) path="/$path" ;;
esac

## It's also useful if you're reading an rcfile and you want to filter out lines starting with a '#'.

Regards,
Thunor
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 3 of 3 [43 Posts]   Goto page: Previous 1, 2, 3
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Off-Topic Area » Programming
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1839s ][ Queries: 12 (0.1038s) ][ GZIP on ]