fields.awk database

[Pete]

Or if you prefer it in pdf format, it's attached.
Just remove the fake .zip file in this case.

I didn't include it in the zip file of the previous post as it took the file size to over the 256KB max.

[musher0]

Thanks, Pete.

[Pete]

You're welcome musher0

Of course it's also just as easy to use geany to convert from fields.awk back to csv.

[cabbie0]

Hello Puppy Linux group,

Delighted to see you folks are finding such ingenious ways to make use of fields.awk.
Keep up the good work!

Just a quick correction in that musher0 somehow got my last name mixed up. It's Forsyth, not Stewart (though they're both Scottish lol).

The work on adaptation to barcode reader outputs and Arduino+sensor data is outstanding creativity. Kudos.

And yes, as you've discovered, using awk in subsequent scripts can produce handy analyses of the fields.awk database files. I do this on a regular basis, especially to keep track of running totals in a given group of items. For example if a simple database for musical items exists like so:

@@@DATE::STORE::ITEMS::TOTAL::
07-16-14::Wentworth Music::violin string set (Helicore 4/4 Light)::50.35::
11-14-14::Wentworth Music::Violin fine tuner (W1102)::4.43::
03-12-15::Wentworth Music::Pro-Arte Violin Set 4/4 (nylon core)::43.63::
05-09-15::London Drugs::Sony Memory Stick Pro Duo (4GB)::33.59::
05-15-15::Wentworth Music::violin string set (D'Addario helicore Heavy-tension 4/4)::50.35::
10-09-15::Wentworth Music::replacement "A" string (Precision A 4/4; 51 Medium, solid steel core, chrome wound)::14.51::
10-16-15::Wentworth Music::violin bow (4/4, carbon composite)::124.27::
10-30-12::Wentworth Music::HOHNER Pro Harp G::44.74::

Then I can enter the following command (of course give it a one-word handle) for the total,

awk -F:: '{print $4}' ~/DB_FILES/music_exp.txt | addcol 1
365.87

Note here the "addcol" command is yet another useful short script based on awk:

# ADDS THE NUMBERS IN COLUMN NUMBER SPECIFIED ON COMMAND LINE
# SYNTAX: addcol [number] [filename]

if [ $# -eq 0 ]
then echo USAGE: addcol [number] [filename]
exit 0
fi

awk -v colnum=$1 'BEGIN {t=0} {t=t+$colnum; print t}' $2 | tail -n1

exit 0

No doubt you may find many uses for that as well. I find it serves very well at tax time, so long as your records are tabulated. No more laborious checking columns with a calculator...


And a huge thank you to Pete for creating the FieldsAwk.png icon Awesome!

Ian R. Forsyth

[greengeek]

Welcome!
I am just getting to grips with fields.awk and have a long way to go...

I enjoyed the youtube video mushero linked to. A good starting point.

I'm hoping to use this to help keep track of tax receipts throughout the year. I think the first hurdle is to decide how many fields to create. The temptation is to create a small number of fields to make it easy to get started - but it may well be better to add more fields than I think I need - in order to allow for future needs.

Or is there a way to add more fields later if necessary? Would that upset prior entries?
.

[Pete]

Hello cabbie0 (Ian) and a warm welcome to you.

Firstly many thanks for creating fields.awk, it's the perfect database, simple, fast, efficient, will work on just about any computer and no dependencies.

Glad you liked the icon, feel free to use it as you see fit.

Handy script "addcol", thanks.

Looking forward to more tips, suggestions and ideas from you.

[musher0]

Welcome aboard, Ian.

I'm very glad that you decided to join the forum. Thanks.

Sorry for confusing your last name with someone else's. (No doubt my most
memorable "oops" ever...) Again, sorry about that.

I hope you enjoy your stay in "the kennels"!

Bye for now.

[Pete]

.... is there a way to add more fields later if necessary? Would that upset prior entries?
.

Two options:

1) Open fields.awk the select (C) Change Editor, by default Ian has coded in Vim if memory serves me right, change it to nano or keep Vim or use another to your liking.
There are several reasons I chose nano, firstly Vim and I are not good friends, nano was already installed and it's CLI based.
Note that you can't use a GUI editor like geany else when you close it, it confuses fields.awk.

EDIT: musher0 has quite rightly pointed out that leafpad does work.

Once you have set the Editor, then you can edit/add/delete fields whilst still in fields.awk

2) If you are not in fields.awk, then you can edit the DB file in a GUI editor if you prefer.

Don't forget the :: at the end of your newly created field/s.

Perhaps Ian has more ideas.

[musher0]

Euh... about GUI editors: I had good results using fields.awk with leafpad. (FWIW)

[Pete]

Aha, and so it does work.
I tried only geany and completely forgot about leafpad.
Thanks musher0, will edit my post above.

[musher0]

Aha, and so it does work.
I tried only geany and completely forgot about leafpad.
Thanks musher0, will edit my post above.

My pleasure, Pete.

Yeah, geany is too cumbersome for such a job.

Back to command-line-interface editors, "joe's own editor" works fine with
fields.awk. On our Puppy forum, the presentation in English is here and the
pet archive is here (at the top of the post, the URL line with "adrive" in it).

I find "joe's own editor" fantastic: it has automatic back-up, can double-up as an
hex editor if you need one, has code-coloring, and can be paired with less --
among many other features. I call it my pseudo-GUI CLI editor!

There's no "line-up at the door" for it, so to speak; still 39 people went to the
trouble of downloading my pet archive of the "joe" editor from adrive.

If you'd like to use "joe" as the editor for fields.awk, 1) install joe from the URL
above ("Gee musher0, how obvious can you be at times!") ; 2) specify
"joe" (without the quotation marks) in fields.awk under the "C" option.

But I'm getting a bit off-topic here. I just thought there was an opportunity to give
this very good "joe" editor a little push! (But nano is great too!)

BFN.

[Pete]

Hi musher0

I was one of the 39 people that downloaded joe.
I do like it and use it as a hex editor.
Only thing that I have against it is the strange keys combinations which takes some getting used to and probably puts a few people off.

[musher0]

That's true.
The combination: Cntrl-K plus another key indeed takes some getting used to.

[cabbie0]

Hi guys.

Regarding greengeek's question about adding new fields _after_ a database has been in use for a while:
Yes, could be done if done carefully by editing the database file itself with the editor of your choice which seems to be a matter of personal preference. I'm hooked on Vim, but realize that many don't care for it. It is very powerful once you get the hang of it but certainly not necessary for fields.awk stuff.
Now, I'd say that deciding carefully in the beginning as to how many fields you want is the better option as it will save much hassle; but should you want to add later I think that adding one or two new fields at the END of the first line would be least disruptive since those new fields would have numbers that would not confuse the earlier entries. If you added a new one somewhere else, then you'd have to add a new " ::" at the appropriate location in all the initial entries to prevent them from lining up the wrong data with the wrong field.

Not to worry musher0 ! I mix up (or rather forget) names all the time. Glad to see it's not me doing it this time he he.

Regarding an earlier concern of greengeek:
" If you try to read some data out of your data file the program asks you if you want to create an output file. If you say yes it will ask you what name to give that file. However - it will not write that file until after you complete the read function and quit back to the home screen. (File is not created till then)."
The reason for this is that is the output from the read function creates the content of the variable $OP, which feeds not only the standard output (to screen) but also the contents of the file to be saved. Prior to the read function happening $OP has no content (null string) and so cannot be used to create a file.

As for the other issue:
" After you create a new datafile you cannot immediately try to add a new record. It is necessary to type "S" first - this gives you an opportunity to select the datafile you want to work with"
This behavior can be altered if you want.
First, I'd like to just mention that originally fields.awk was simply used from the command line and the database file was specified in the launch command. As in: "fields.awk [path_to_datafile]" and the relevant file was selected. This behavior still works (i.e. backward compatible); you can try it. It works because the very first line of the script is still:
DATAFILE=$1; touch ~/tmp ~/output; TMP=~/tmp; OP=~/output
Note the first part that defines DATAFILE as $1 which in this case is the first option entered after the command itself.
The new default, where fields.awk is simply launched empty and all the datafiles are collected in a central repository from which they are selected by the "S" option, results in the variable "DATAFILE" initially being empty and therefore nothing can be done till one is selected.

However, in your special case of creating a new database file from the "N" option, even though it's only done once I suppose you're right that it would be more convenient if it were automatically selected immediately upon creation.
So, if you want to do that, go to the "new_data" function section of the script, near the end of the function where you see the line:
mv -f dbfile.tmp $DB_FOLDER/$NEW_DATAFILE (line 261 or thereabouts...)

Immediately after this line add:
DATAFILE=$DB_FOLDER/$NEW_DATAFILE
Then edit the output line,
echo "Your new database_file should be ready to select."
to read instead:
echo "Your new database_file is created and currently selected."
and either delete or comment out the line:
echo "Current contents of $DB_FOLDER folder: "; echo `ls -m $DB_FOLDER`
because you are no longer selecting from the list of files in the repository.

I tried this; it works. But of course always be careful editing a script as very small syntax issues can cause weird behavior.

That's about all I've got for now
Have fun.

[greengeek]

Now, I'd say that deciding carefully in the beginning as to how many fields you want is the better option as it will save much hassle; but should you want to add later I think that adding one or two new fields at the END of the first line would be least disruptive since those new fields would have numbers that would not confuse the earlier entries.

Thanks. Good idea. I think I will make it a habit to add a couple of extra fields at the end. My last 3 fields will be Comments, Comments2 and Comments3 and I think I should be ok to rename the last two fields later if required.

...it will not write that file until after you complete the read function and quit back to the home screen. (File is not created till then)."
The reason for this is that is the output from the read function creates the content of the variable $OP, which feeds not only the standard output (to screen) but also the contents of the file to be saved. Prior to the read function happening $OP has no content (null string) and so cannot be used to create a file.

Thanks for the explanation. Now that I've seen how it works it's easy to use.

" After you create a new datafile you cannot immediately try to add a new record. It is necessary to type "S" first - this gives you an opportunity to select the datafile you want to work with"
This behavior can be altered if you want.....
I tried this; it works. But of course always be careful editing a script as very small syntax issues can cause weird behavior.

Thanks very much for the mod - that makes it even more useable. I think it is better for newbies this way - when I first tried it I spent quite a lot of time wondering how to make it work (obviously missed the top line prompt in the terminal) but in fact the programme is quite self-explanatory and I think this change makes it a no-brainer for the first time user. All straightforward.
cheers!

[Pete]

@cabbie0

Thanks Ian, that's a handy modification.

[Pete]

Several posts back, I included a one liner to convert a fields.awk DB to html.
It was limited in that in only converted to straight html with no regards for links.

One of my uses of fields.awk is to enter bits of info into a database so I can find them again.
It's a simple database with only 3 fields, Name URL and comments.
Here is an excerpt:

Code: Select all

@@@Name::URL::Comments::
Google::http://www.google.com::search engine::
Puppy::http://www.murga-linux.com/puppy/index.php::Puppy forum index::
Puppy::http://www.murga-linux.com/puppy/viewtopic.php?p=909228::fields.awk thread::
Puppy::http://www.murga-linux.com/puppy/index.php?f=63::Puppt additional pets::
Fields.awk::https://www.youtube.com/embed/HC8Az9W8FV4?rel=0&autoplay=0::fields.awk video::

What I needed was a converter to view this in a web browser so I can click directly on the URLs and go to that web site.
So I came up with a Bash script called fields2html, attached, just remove the fake .zip and don't forget to set permissions.

Now what I do is open fields.awk, select my database described above then do a search/read, saving it to a file, not case sensitive then just pressing enter at the search criteria so that fields.awk will include everything.

Now I run that file generated by fields.awk (search1.txt in /root/.DB) thru fields2html:

Code: Select all

fields2html /root/.DB/search1.txt

Result attached as a screenshot.
Note also that by clicking on a link, it opens in a new tab leaving the original tab still containing the list so that one can open multiple web sites instead of having to click on BACK all time.

Some notes/gotchas:

1) Don't parse the fields.awk DB directly to fields2html as the :: causes the links to become malformed, rather use the file created by fields.awk from a search/read.
Of course if you don't have any URLs in you database, then go for it.

2) The script also gives the user the option to open the newly created html file in the defaultbrowser.
If you are not running Puppy or prefer to open another web browser, modify the last line of the script accordingly.

3) The script creates a html file with the same name (plus .html extension) in the same directory as the input file.

Don't forget to remove the fake .zip in the attached script below.

Lastly, I decided to call the script fields2html and not text2html (which it technically is) as some of you may already have a script/pgm called that.

[greengeek]

So I came up with a Bash script called fields2html,...
Now what I do is open fields.awk, select my database described above then do a search/read, saving it to a file, not case sensitive then just pressing enter at the search criteria so that fields.awk will include everything.

Now I run that file generated by fields.awk (search1.txt in /root/.DB) thru fields2html:

Code: Select all

fields2html /root/.DB/search1.txt

Wow, thats a handy function! Given that ian says fields.awk can be cli driven I wonder if it would be possible to "pipe" the output of fields.awk direct to fields2html without manual manipulation of the database search?

I can't code to save my life but something like this? -

Code: Select all

fields.awk [path_to_datafile] readall "*" /root/searchresult.txt | fields2html /root/searchresult.txt > links_out.html

or:

Code: Select all

fields.awk [path_to_datafile] readall "*" | fields2html > links_out.html

The only thing missing from my example is accuracy and correct syntax

.

[musher0]

@Pete: that is indeed a handy addition!

[Pete]

@musher0
Thanks

@greengeek

Is this what you meant?

Version 2 of fields2html attached, remove fake .zip
Just delete the previous version and replace with this one.

One can now convert a fields.awk database directly without using fields,awk or any manipulation.

Code: Select all

fields2html /path/to/database.txt

Of course you can also use the search file created by fields.awk if you wish.