Odd Bash script inconsistancy...

For discussions about programming, programming questions/advice, and projects that don't really have anything to do with Puppy.
Post Reply
Message
Author
User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

Odd Bash script inconsistancy...

#1 Post by sunburnt »

I wrote a little script to add up the size of files selected by wildcard argument.
The lines of code work fine run individually in Xterm.
But in the script it only gets the first file, not the whole wildcard list.

Code: Select all

#!/bin/sh
#####	Get total size of files by wildcard argument.

if [ -z "$1" ];then exit ;fi
FILES=`du $1`
#FILES=`ls -1l $1 | awk '{print $5, $8}'`
echo '###   FILES'
echo "$FILES"
SIZES=`echo "$FILES" | awk '{print $1}'`
echo '###   SIZES'
echo "$SIZES"
ADD=`echo $SIZES |  sed 's/ / + /g'`
echo '###   $ADD'
echo "$ADD"
echo '============================================='
echo "$FILES"
echo '================================'
echo `expr $ADD`' Total of File Sizes.'
The bad lines are "ls" or "du", and both give only the first line in the script.
If you run either line in Xterm it gives all the matching files and sizes.
Last edited by sunburnt on Fri 22 Jan 2010, 20:10, edited 1 time in total.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#2 Post by amigo »

Instead of calling ls, awk(twice!), sed, du and expr:

Code: Select all

#!/bin/bash
for FILE in * ; do
	[[ -f $FILE ]] && TOTAL_SIZE=$(( $TOTAL_SIZE + $(stat -c %s $FILE) ))
done
echo TOTAL_SIZE=$TOTAL_SIZE

seaside
Posts: 934
Joined: Thu 12 Apr 2007, 00:19

#3 Post by seaside »

Also, how about this -

Code: Select all

ls -l | awk  '{total+=$5} END{print total}'

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#4 Post by sunburnt »

Thanks guys! But...

amigo; Your code also only returns the first file! ( run in a Bash script )
I changed the * to $1 so the wild card path would work properly.

Code: Select all

for FILE in $1 ; do
   [[ -f $FILE ]] && TOTAL_SIZE=$(( $TOTAL_SIZE + $(stat -c %s $FILE) ))
   echo $FILE
done
echo TOTAL_SIZE=$TOTAL_SIZE ;exit
It returns: TOTAL_SIZE=1607264396 ( the size of the first file )
There are 2 files in the dir. that start with C*
The line "echo $FILE" shows this to be true.

seaside; I`m not sure how your code is to be used...
You use $5 to get the file sizes from "ls", but "ls" is the problem !!!
Again... All of this works in a console, but it doesn`t in a Bash script.

### My hope was to display the files and their sizes, and a total.
That`s why the many lines in my code to get all the display items.
The other lines are to test each step to show where the failure is.

This is just another example of the incongruity of Bash and the other commands that support it.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#5 Post by amigo »

I originally had this:
for FILE in $(ls -1) ; do
but any files with spces in the names would not work there.
The '*' works fine here and should anywhere -very strange if it is not working for you.

seaside
Posts: 934
Joined: Thu 12 Apr 2007, 00:19

Re: Odd Bash script inconsistancy...

#6 Post by seaside »

sunburnt wrote:I wrote a little script to add up the size of files selected by wildcard argument.
Sunburnt,

I thought you might just use "ls" with wildcard selection like this (not in a loop).

Code: Select all

ls -l *wildcard* | awk  '{total+=$5} END{print total}'
s.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#7 Post by sunburnt »

amigo; Try it like this so a wildcard argument can be fed to it:

Code: Select all

#!/bin/sh
#####	Get total size of files by wildcard argument.
if [ -z "$1" ];then exit ;fi
for FILE in $1 ; do
  echo $FILE
done
Let me know if it works for you and gives the full wildcard list.

seaside; That is what my script does...
My code line:

Code: Select all

FILES=`ls -1l $1 | awk '{print $5, $8}'`
Your code line:

Code: Select all

ls -l *wildcard* | awk  '{total+=$5} END{print total}'
Have you tried using "ls" in a script to see if it will list all the wildcard files?
Both "ls" and "du" seem to work at the command line but not in a script.
All I get from them and amigo`s "for loop" is the first wildcard match, not the list.

seaside
Posts: 934
Joined: Thu 12 Apr 2007, 00:19

#8 Post by seaside »

sunburnt,

This works in a script.

Code: Select all

#!/bin/sh
wild=s*
total=`ls -l $wild | awk  '{total+=$5} END{print total}'`
echo $total
files=`ls -l *.txt`
echo $files

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#9 Post by sunburnt »

seaside; I don`t know if it`s the fact I need to use $1 for the wildcard argument.
Again... Your code only pulls the first file and it`s size! ( using $1 )

Code: Select all

#!/bin/sh
if [ -z "$1" ];then exit ;fi
wild=$1
total=`ls -l $wild | awk  '{total+=$5} END{print total}'`
echo $total
files=`ls -l $1`
echo $files
Using "ls", "du", and "stat" just doesn`t seem to work in a script...
Try the simplest form:

Code: Select all

#!/bin/sh
if [ -z "$1" ];then exit ;fi
ls -l1 $1
du $1
All methods ( first "ls" then "du" ) give this:

Code: Select all

# filesize /mnt/sda6/v/0/C*
-rw-r--r-- 1 root root 1607264396 2010-01-07 06:32 /mnt/sda6/v/0/Children-1.jpg
1607264396 /mnt/sda6/v/0/Children-1.jpg
There are the pix: /mnt/sda6/0/Children-1.jpg and /mnt/sda6/0/Children-2.jpg

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#10 Post by amigo »

ls and du both can cause problems because they use tabs to format the output -maybe depending on whether run in a script. du is particularly troublesome.

The script fragment I posted is *bash* not POSIX shell, so /bin/sh may not work for the shebang -depending on what /bin/sh is linked to.

There is no way that something like this:
wild=$1
total=`ls -l $wild
is gonna return more than one item, unless you do something like echo * | name-of-script

This also works:
for FILE in $(echo *) ; do

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#11 Post by sunburnt »

amigo; I tried it with "#!/bin/bash", no go... I didn`t think it`d work.
The "for" loop only iterates just once using "$1".
However... Putting the argument in place of "$1" works!
So... Using the "$1" argument variable seem to mess it up!!!
Try this:

Code: Select all

#!/bin/sh
echo "$1"
It gives the first file... Not the wildcard argument.!!!
It seems that the wildcard is being interpreted by being the "$1" argument!
So feeding wildcards to a script just doesn`t work! Period!
This works with: /mnt/sda6/0/C
### Sort of...

Code: Select all

#!/bin/sh
echo "$1"*
You have to assume the "*" at the end, it can`t be placed in the argument...
But it keeps the argument from being interpreted, so it works.

User avatar
sunburnt
Posts: 5090
Joined: Wed 08 Jun 2005, 23:11
Location: Arizona, U.S.A.

#12 Post by sunburnt »

So "ls", "du". or any other command isn`t the problem, it`s that
the wildcard argument is interpreted and it can`t be stopped.

Here`s what I came up with, it`s not what I wanted ( as usual...).
The script adds the "*" on the end of the argument:

Code: Select all

#!/bin/sh
#####	Get total size of files by wildcard argument.
if [ -z "$1" ];then exit ;fi
FILES=`du "$1"*`
SIZES=`echo "$FILES" | awk '{print $1}'`
ADD=`echo $SIZES |  sed 's/ / + /g'`
echo '==============================================='
echo "$FILES"
echo '============================'
echo `expr $ADD`' Total of File Sizes.'
So you use "/mnt/sda6/0/C" instead of "/mnt/sda6/0/C*"
As before, if you add the "*" at the end it`ll only give the first file.

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#13 Post by jpeps »

maybe more useful:

Code: Select all

FILES=`du -S  "$1"*` 

big_bass
Posts: 1740
Joined: Mon 13 Aug 2007, 12:21

#14 Post by big_bass »

I used this in one of my scripts

Code: Select all

#find total size 
filesizes=`du -ch | grep total`
echo $filesizes


here was the original script it creates an index
I wrote it to make the index for my packages

when you dragNdrop a folder on it

it works fine if you dont have folders in subdirectories
I still need to figure out how to get that working correctly
if you amigo or anyone else gets the subdirectories working
it would make for a nice file browser too :D

Code: Select all


#!/bin/sh
# last updated 2-3-2010
# script  made by Joe arose first template  
# this makes the index.html for a simple package repo
# it is an all in one script so I can easily keep the package list updated 
# I built this out of need and it needed to be easily updated 
# I did'nt want web page developement to take all my time 
# Im too busy doing other things 


-------------------

# 
# you dragN drop a folder on this script
# or set up the right click in ROX

# the rxvt terminal opens there
# you could change rxvt to xterm if you prefer
# Joe Arose ...big_bass 

DIR="$1"
 
if [ -d "$DIR" ]
then
   echo "$DIR directory  exists!"
   cd "$1"
  ##  rxvt >/dev/null 2>&1
else
   echo "$DIR directory not found!"
    exec xmessage -bg "yellow" -center -title "ERROR" "folders /directories only " 
fi 


-----------------


echo >/root/html-tagset.html  #blank it  will be renamed to index.php-new later  
cd $DIR  

#find total folder size 
filesizes=`du -ch | grep total`






# starts the head of the html  edit the names its a big echo
echo '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /slaxer_pup 4.12 </title>
 </head></font>
 <body>
<BODY BGCOLOR="#000000" BACKGROUND="stars_background.gif">
 <pre><img src="slxminilogo.gif" alt="Icon "> <a href="?C=N;O=D"></a>   
 <pre><img src="newlogo.png" alt="Icon "> <a href="?C=N;O=D"></a><br />
<font size="+2" color="#FFFFFF"> </a>
<br />'>>/root/html-tagset.html 

#end of the big echo and the end the html head 

# here below  is a simple sample format of what the output will look like without the  package size and date

#<img src=compressed.png alt=[ ]> <a href=geany-0.17-i486-slxr.pet><font size="5" color="#FFFF00">geany-0.17-i486-slxr.pet</font></a>


# body of the  html code is generated by bash 
# this was tricky to get the readable values 
#in the format for package name  date and size 
#to get everything read at once for each package  


for package in `ls -p `; do
name=`basename "$package"`
datemodified=`ls -lh "$package" | awk '{ print $6 }'`
size=`ls -lh "$package" | awk '{ print $5 }'` 


# edit this with care it was tedious to format it. Save a copy first  
echo "<img src="compressed.png" alt="[  ]">" "<"a href="$name"">""<"font size="5" color="#FFFF00"">"$name"<"/font">""<"/a">" "<"font size="5" color="#FFFFFF"">"  $datemodified "<"font size="5" color="#FFFF00"">"   $size >>/root/html-tagset.html        

done

# the last echo to end the index 
echo "<hr></pre>$filesizes

</body></html>">>/root/html-tagset.html    

mv /root/html-tagset.html  /root/index.php-new
rm /root/html-tagset.html 
if you copy
index.php-new that file into the folder you draNdropped it will work as an indexer


updated again
I fixed it to do the subdirectories
for package in `ls -p `; do
Joe
Last edited by big_bass on Thu 04 Feb 2010, 20:58, edited 4 times in total.

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#15 Post by amigo »

I seem to have mis-read the original intention of the script.
First, you are wanting to pass arguments to this script to specify one or more locations and/or types of files?
And you want to be able to use wildcards in those arguments?

Answering in order:
if you want to pass multiple args to the script, then using '$1' is not right. You need to use $@ so that all the arguments get interpreted (by a loop):

Code: Select all

#!/bin/sh
for ITEM in $@ ; do
 echo $ITEM
done
I assume you may be wanting to process directories instead of just files, so this will work for dirs as well:

Code: Select all

#!/bin/sh
for ITEM in $@ ; do
	SIZE=$(du -sk $ITEM)
	SIZE=$(echo $SIZE |cut -f1 -d' ')
	TOTAL_SIZE=$(( $TOTAL_SIZE + $SIZE ))
	# the next line here is for debugging
	echo $ITEM = $SIZE
done
echo TOTAL_SIZE = $TOTAL_SIZE KB
Now about wildcards. They are going to be expanded by the shell *before* passing them to the script. for instance, if you save th above script as 'tryme' and place it in a directory where there are several files with a '.txt' suffix, if you run:
sh tryme *.txt
it will work correctly in that directory. It will also work like this:
sh tryme *.txt test/*.txt /path/to/another/plain/dir
(To show how the shell expands these first, just do:
echo *.txt test/*.txt
or whatever paths with wildcards you are using)

Here you have to use 'du' to get the sizes for directories as stat will not do that. You need the '-s' option to output just the summary total for the item, and the '-k' option so you know hat the unit is. If you use the '-h', then you'll get mixed output with some items as ??K, some as ??M and whatever.

You have to do this:
SIZE=$(echo $SIZE |cut -f1 -d' ')
so that echo reduces the whitespaces in the output from du. When you echo something unquoted, any mutiple spaces get converted to single spaces, tabs get converted to single spaces and any leading spaces are eliminated. The output from du is formatted with tabs or maybe even a mixture of tabs and spaces, so if you tried this:
SIZE=$(du -sk $ITEM|cut -f1 -d' ')
you'll get an error because of the garbage it returns(as fas as doing any mat is concerned)

This will work:
SIZE=$(echo $(du -sk $ITEM)|cut -f1 -d' ')
but this will not:
SIZE=$(echo $(du -sk $ITEM|cut -f1 -d' '))
I do them separately in the example for better dependability and readability. Since 'echo' is a shell builtin it doesn't cost any extra *external* process to do it.
awk will handle the tabs/spaces problem handily enough, but I still think it is overkill for this task. And, you could use ls instead of 'du', but you'll have the same problem with formatted output, and if you wanto to return totals from content inside directories, then you'd have to make an extra loop which can descend(mutiple levels) into each dir/subdir and add up each item inside there. 'du' already does that for you, and recursing routines with the sehll get messy.

Of course, when you are done, you can rename the script to whatever, make it executable and drop it in your PATH to be able to use it from wherever.

seaside
Posts: 934
Joined: Thu 12 Apr 2007, 00:19

#16 Post by seaside »

sunburnt wrote: It gives the first file... Not the wildcard argument.!!!
It seems that the wildcard is being interpreted by being the "$1" argument!
So feeding wildcards to a script just doesn`t work! Period!
Well isn't that curious!

Code: Select all

./script "t*"
WORKS

so you would think that if you build a wildcard parameter with single quotes around it - like this -

Code: Select all

# w='"t*"'
# echo $w
"t*"
# 
would do the trick - but no -

Code: Select all

./script $w or "$w"
FAILS

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#17 Post by amigo »

Enclosing in single quotes turns off expansion of wildcards -i mean the outer quotes. This: "'$var'" would still get expanded. Sometimes you have to escape characters to keep them from being immediately expanded by the shell: \$var

seaside
Posts: 934
Joined: Thu 12 Apr 2007, 00:19

#18 Post by seaside »

amigo,

Thanks for the explanation. $@ works nicely. Also found a good short description and chart of the differences between $* and $@ here -
http://fvue.nl/wiki/Bash:_Difference_between_$@_and_$*

As a general rule they suggest mostly using $@

s

Post Reply