| Author |
Message |
sunburnt
Joined: 08 Jun 2005
Posts: 5082
Location: Arizona, U.S.A.
|
 Posted: Fri 22 Jan 2010, 15:06 Post subject:
 Odd Bash script inconsistancy... |
|
I wrote a little script to add up the size of files selected by wildcard argument.
The lines of code work fine run individually in Xterm.
But in the script it only gets the first file, not the whole wildcard list.
| Code: | #!/bin/sh
##### Get total size of files by wildcard argument.
if [ -z "$1" ];then exit ;fi
FILES=`du $1`
#FILES=`ls -1l $1 | awk '{print $5, $8}'`
echo '### FILES'
echo "$FILES"
SIZES=`echo "$FILES" | awk '{print $1}'`
echo '### SIZES'
echo "$SIZES"
ADD=`echo $SIZES | sed 's/ / + /g'`
echo '### $ADD'
echo "$ADD"
echo '============================================='
echo "$FILES"
echo '================================'
echo `expr $ADD`' Total of File Sizes.' |
The bad lines are "ls" or "du", and both give only the first line in the script.
If you run either line in Xterm it gives all the matching files and sizes.
Last edited by sunburnt on Fri 22 Jan 2010, 16:10; edited 1 time in total
|
|
Back to top
|
|
 |
amigo
Joined: 02 Apr 2007
Posts: 2544
|
| Posted: Fri 22 Jan 2010, 15:34 Post subject:
|
|
Instead of calling ls, awk(twice!), sed, du and expr:
| Code: | #!/bin/bash
for FILE in * ; do
[[ -f $FILE ]] && TOTAL_SIZE=$(( $TOTAL_SIZE + $(stat -c %s $FILE) ))
done
echo TOTAL_SIZE=$TOTAL_SIZE
|
|
|
Back to top
|
|
|
seaside
Joined: 11 Apr 2007
Posts: 911
|
| Posted: Fri 22 Jan 2010, 20:47 Post subject:
|
|
Also, how about this -
| Code: | | ls -l | awk '{total+=$5} END{print total}' |
|
|
Back to top
|
|
|
sunburnt
Joined: 08 Jun 2005
Posts: 5082
Location: Arizona, U.S.A.
|
| Posted: Sat 23 Jan 2010, 16:05 Post subject:
|
|
Thanks guys! But...
amigo; Your code also only returns the first file! ( run in a Bash script )
I changed the * to $1 so the wild card path would work properly.
| Code: | for FILE in $1 ; do
[[ -f $FILE ]] && TOTAL_SIZE=$(( $TOTAL_SIZE + $(stat -c %s $FILE) ))
echo $FILE
done
echo TOTAL_SIZE=$TOTAL_SIZE ;exit |
It returns: TOTAL_SIZE=1607264396 ( the size of the first file )
There are 2 files in the dir. that start with C*
The line "echo $FILE" shows this to be true.
seaside; I`m not sure how your code is to be used...
You use $5 to get the file sizes from "ls", but "ls" is the problem !!!
Again... All of this works in a console, but it doesn`t in a Bash script.
### My hope was to display the files and their sizes, and a total.
That`s why the many lines in my code to get all the display items.
The other lines are to test each step to show where the failure is.
This is just another example of the incongruity of Bash and the other commands that support it.
|
|
Back to top
|
|
|
amigo
Joined: 02 Apr 2007
Posts: 2544
|
| Posted: Sat 23 Jan 2010, 16:27 Post subject:
|
|
I originally had this:
for FILE in $(ls -1) ; do
but any files with spces in the names would not work there.
The '*' works fine here and should anywhere -very strange if it is not working for you.
|
|
Back to top
|
|
|
seaside
Joined: 11 Apr 2007
Posts: 911
|
| Posted: Sat 23 Jan 2010, 17:24 Post subject:
Re: Odd Bash script inconsistancy... |
|
| sunburnt wrote: | I wrote a little script to add up the size of files selected by wildcard argument.
|
Sunburnt,
I thought you might just use "ls" with wildcard selection like this (not in a loop).
| Code: | | ls -l *wildcard* | awk '{total+=$5} END{print total}' |
s.
|
|
Back to top
|
|
|
sunburnt
Joined: 08 Jun 2005
Posts: 5082
Location: Arizona, U.S.A.
|
| Posted: Sat 23 Jan 2010, 21:04 Post subject:
|
|
amigo; Try it like this so a wildcard argument can be fed to it:
| Code: | #!/bin/sh
##### Get total size of files by wildcard argument.
if [ -z "$1" ];then exit ;fi
for FILE in $1 ; do
echo $FILE
done |
Let me know if it works for you and gives the full wildcard list.
seaside; That is what my script does...
My code line:
| Code: | | FILES=`ls -1l $1 | awk '{print $5, $8}'` |
Your code line:
| Code: | | ls -l *wildcard* | awk '{total+=$5} END{print total}' |
Have you tried using "ls" in a script to see if it will list all the wildcard files?
Both "ls" and "du" seem to work at the command line but not in a script.
All I get from them and amigo`s "for loop" is the first wildcard match, not the list.
|
|
Back to top
|
|
|
seaside
Joined: 11 Apr 2007
Posts: 911
|
| Posted: Sun 24 Jan 2010, 00:12 Post subject:
|
|
sunburnt,
This works in a script.
| Code: | #!/bin/sh
wild=s*
total=`ls -l $wild | awk '{total+=$5} END{print total}'`
echo $total
files=`ls -l *.txt`
echo $files |
|
|
Back to top
|
|
|
sunburnt
Joined: 08 Jun 2005
Posts: 5082
Location: Arizona, U.S.A.
|
| Posted: Sun 24 Jan 2010, 13:19 Post subject:
|
|
seaside; I don`t know if it`s the fact I need to use $1 for the wildcard argument.
Again... Your code only pulls the first file and it`s size! ( using $1 )
| Code: | #!/bin/sh
if [ -z "$1" ];then exit ;fi
wild=$1
total=`ls -l $wild | awk '{total+=$5} END{print total}'`
echo $total
files=`ls -l $1`
echo $files |
Using "ls", "du", and "stat" just doesn`t seem to work in a script...
Try the simplest form:
| Code: | #!/bin/sh
if [ -z "$1" ];then exit ;fi
ls -l1 $1
du $1 |
All methods ( first "ls" then "du" ) give this:
| Code: | # filesize /mnt/sda6/v/0/C*
-rw-r--r-- 1 root root 1607264396 2010-01-07 06:32 /mnt/sda6/v/0/Children-1.jpg
1607264396 /mnt/sda6/v/0/Children-1.jpg |
There are the pix: /mnt/sda6/0/Children-1.jpg and /mnt/sda6/0/Children-2.jpg
|
|
Back to top
|
|
|
amigo
Joined: 02 Apr 2007
Posts: 2544
|
| Posted: Sun 24 Jan 2010, 14:06 Post subject:
|
|
ls and du both can cause problems because they use tabs to format the output -maybe depending on whether run in a script. du is particularly troublesome.
The script fragment I posted is *bash* not POSIX shell, so /bin/sh may not work for the shebang -depending on what /bin/sh is linked to.
There is no way that something like this:
wild=$1
total=`ls -l $wild
is gonna return more than one item, unless you do something like echo * | name-of-script
This also works:
for FILE in $(echo *) ; do
|
|
Back to top
|
|
|
sunburnt
Joined: 08 Jun 2005
Posts: 5082
Location: Arizona, U.S.A.
|
| Posted: Sun 24 Jan 2010, 14:35 Post subject:
|
|
amigo; I tried it with "#!/bin/bash", no go... I didn`t think it`d work.
The "for" loop only iterates just once using "$1".
However... Putting the argument in place of "$1" works!
So... Using the "$1" argument variable seem to mess it up!!!
Try this:
| Code: | #!/bin/sh
echo "$1" |
It gives the first file... Not the wildcard argument.!!!
It seems that the wildcard is being interpreted by being the "$1" argument!
So feeding wildcards to a script just doesn`t work! Period!
This works with: /mnt/sda6/0/C
### Sort of...
| Code: | #!/bin/sh
echo "$1"* |
You have to assume the "*" at the end, it can`t be placed in the argument...
But it keeps the argument from being interpreted, so it works.
|
|
Back to top
|
|
|
sunburnt
Joined: 08 Jun 2005
Posts: 5082
Location: Arizona, U.S.A.
|
| Posted: Sun 24 Jan 2010, 20:48 Post subject:
|
|
So "ls", "du". or any other command isn`t the problem, it`s that
the wildcard argument is interpreted and it can`t be stopped.
Here`s what I came up with, it`s not what I wanted ( as usual...).
The script adds the "*" on the end of the argument:
| Code: | #!/bin/sh
##### Get total size of files by wildcard argument.
if [ -z "$1" ];then exit ;fi
FILES=`du "$1"*`
SIZES=`echo "$FILES" | awk '{print $1}'`
ADD=`echo $SIZES | sed 's/ / + /g'`
echo '==============================================='
echo "$FILES"
echo '============================'
echo `expr $ADD`' Total of File Sizes.' |
So you use "/mnt/sda6/0/C" instead of "/mnt/sda6/0/C*"
As before, if you add the "*" at the end it`ll only give the first file.
|
|
Back to top
|
|
|
jpeps
Joined: 31 May 2008
Posts: 3217
|
| Posted: Mon 25 Jan 2010, 02:58 Post subject:
|
|
maybe more useful:
| Code: |
FILES=`du -S "$1"*`
|
|
|
Back to top
|
|
|
big_bass
Joined: 13 Aug 2007
Posts: 1742
|
| Posted: Mon 25 Jan 2010, 13:03 Post subject:
|
|
I used this in one of my scripts
| Code: | #find total size
filesizes=`du -ch | grep total`
echo $filesizes |
here was the original script it creates an index
I wrote it to make the index for my packages
when you dragNdrop a folder on it
it works fine if you dont have folders in subdirectories
I still need to figure out how to get that working correctly
if you amigo or anyone else gets the subdirectories working
it would make for a nice file browser too 
| Code: |
#!/bin/sh
# last updated 2-3-2010
# script made by Joe arose first template
# this makes the index.html for a simple package repo
# it is an all in one script so I can easily keep the package list updated
# I built this out of need and it needed to be easily updated
# I did'nt want web page developement to take all my time
# Im too busy doing other things
-------------------
#
# you dragN drop a folder on this script
# or set up the right click in ROX
# the rxvt terminal opens there
# you could change rxvt to xterm if you prefer
# Joe Arose ...big_bass
DIR="$1"
if [ -d "$DIR" ]
then
echo "$DIR directory exists!"
cd "$1"
## rxvt >/dev/null 2>&1
else
echo "$DIR directory not found!"
exec xmessage -bg "yellow" -center -title "ERROR" "folders /directories only "
fi
-----------------
echo >/root/html-tagset.html #blank it will be renamed to index.php-new later
cd $DIR
#find total folder size
filesizes=`du -ch | grep total`
# starts the head of the html edit the names its a big echo
echo '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /slaxer_pup 4.12 </title>
</head></font>
<body>
<BODY BGCOLOR="#000000" BACKGROUND="stars_background.gif">
<pre><img src="slxminilogo.gif" alt="Icon "> <a href="?C=N;O=D"></a>
<pre><img src="newlogo.png" alt="Icon "> <a href="?C=N;O=D"></a><br />
<font size="+2" color="#FFFFFF"> </a>
<br />'>>/root/html-tagset.html
#end of the big echo and the end the html head
# here below is a simple sample format of what the output will look like without the package size and date
#<img src=compressed.png alt=[ ]> <a href=geany-0.17-i486-slxr.pet><font size="5" color="#FFFF00">geany-0.17-i486-slxr.pet</font></a>
# body of the html code is generated by bash
# this was tricky to get the readable values
#in the format for package name date and size
#to get everything read at once for each package
for package in `ls -p `; do
name=`basename "$package"`
datemodified=`ls -lh "$package" | awk '{ print $6 }'`
size=`ls -lh "$package" | awk '{ print $5 }'`
# edit this with care it was tedious to format it. Save a copy first
echo "<img src="compressed.png" alt="[ ]">" "<"a href="$name"">""<"font size="5" color="#FFFF00"">"$name"<"/font">""<"/a">" "<"font size="5" color="#FFFFFF"">" $datemodified "<"font size="5" color="#FFFF00"">" $size >>/root/html-tagset.html
done
# the last echo to end the index
echo "<hr></pre>$filesizes
</body></html>">>/root/html-tagset.html
mv /root/html-tagset.html /root/index.php-new
rm /root/html-tagset.html
|
if you copy
index.php-new that file into the folder you draNdropped it will work as an indexer
updated again
I fixed it to do the subdirectories
for package in `ls -p `; do
Joe
Last edited by big_bass on Thu 04 Feb 2010, 16:58; edited 4 times in total
|
|
Back to top
|
|
|
amigo
Joined: 02 Apr 2007
Posts: 2544
|
| Posted: Mon 25 Jan 2010, 14:56 Post subject:
|
|
I seem to have mis-read the original intention of the script.
First, you are wanting to pass arguments to this script to specify one or more locations and/or types of files?
And you want to be able to use wildcards in those arguments?
Answering in order:
if you want to pass multiple args to the script, then using '$1' is not right. You need to use $@ so that all the arguments get interpreted (by a loop):
| Code: | #!/bin/sh
for ITEM in $@ ; do
echo $ITEM
done
|
I assume you may be wanting to process directories instead of just files, so this will work for dirs as well:
| Code: | #!/bin/sh
for ITEM in $@ ; do
SIZE=$(du -sk $ITEM)
SIZE=$(echo $SIZE |cut -f1 -d' ')
TOTAL_SIZE=$(( $TOTAL_SIZE + $SIZE ))
# the next line here is for debugging
echo $ITEM = $SIZE
done
echo TOTAL_SIZE = $TOTAL_SIZE KB
|
Now about wildcards. They are going to be expanded by the shell *before* passing them to the script. for instance, if you save th above script as 'tryme' and place it in a directory where there are several files with a '.txt' suffix, if you run:
sh tryme *.txt
it will work correctly in that directory. It will also work like this:
sh tryme *.txt test/*.txt /path/to/another/plain/dir
(To show how the shell expands these first, just do:
echo *.txt test/*.txt
or whatever paths with wildcards you are using)
Here you have to use 'du' to get the sizes for directories as stat will not do that. You need the '-s' option to output just the summary total for the item, and the '-k' option so you know hat the unit is. If you use the '-h', then you'll get mixed output with some items as ??K, some as ??M and whatever.
You have to do this:
SIZE=$(echo $SIZE |cut -f1 -d' ')
so that echo reduces the whitespaces in the output from du. When you echo something unquoted, any mutiple spaces get converted to single spaces, tabs get converted to single spaces and any leading spaces are eliminated. The output from du is formatted with tabs or maybe even a mixture of tabs and spaces, so if you tried this:
SIZE=$(du -sk $ITEM|cut -f1 -d' ')
you'll get an error because of the garbage it returns(as fas as doing any mat is concerned)
This will work:
SIZE=$(echo $(du -sk $ITEM)|cut -f1 -d' ')
but this will not:
SIZE=$(echo $(du -sk $ITEM|cut -f1 -d' '))
I do them separately in the example for better dependability and readability. Since 'echo' is a shell builtin it doesn't cost any extra *external* process to do it.
awk will handle the tabs/spaces problem handily enough, but I still think it is overkill for this task. And, you could use ls instead of 'du', but you'll have the same problem with formatted output, and if you wanto to return totals from content inside directories, then you'd have to make an extra loop which can descend(mutiple levels) into each dir/subdir and add up each item inside there. 'du' already does that for you, and recursing routines with the sehll get messy.
Of course, when you are done, you can rename the script to whatever, make it executable and drop it in your PATH to be able to use it from wherever.
|
|
Back to top
|
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index » Off-Topic Area » Programming
