fields.awk database

Under development: PCMCIA, wireless, etc.
Message
Author
User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#76 Post by Pete »

So now we know that fields.awk will also work under:

- MAC (OSX)
- BSD
- Android (with Term emulator)
- Windows (only tried 8.1 but no reason to believe that it won't work under 7 and XP).

Now only the Raspberry Pie left.

Perhaps others too like the Banana Pie and Beagle Board but I know nothing about them.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Cross-platform fields.awk

#77 Post by cabbie0 »

@Pete

I never thought I'd see fields.awk running on a Windows machine.
Wonders never cease.
It makes me curious how well/poorly it might be received by that new audience.

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

Re: Cross-platform fields.awk

#78 Post by Pete »

cabbie0 wrote:@Pete

I never thought I'd see fields.awk running on a Windows machine.
Wonders never cease.
I have to admit that seeing the windows logo and things like /usr/sbin and awk in the same screenshot is a bit strange. :lol:
It makes me curious how well/poorly it might be received by that new audience.
Good question.
My view is that Windows users may be even more reluctant than Linux ones when it comes to using the CLI.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Search Output processing

#79 Post by cabbie0 »

My view is that Windows users may be even more reluctant than Linux ones when it comes to using the CLI.
@Pete
That's my gut feeling as well. Still it's interesting to see how adaptable it is to various versions of the bash shell environment.

@all
greengeek earlier brought up the issue of printing out a read search file containing ALL the records of a given database.
And I answered that hitting the ENTER key at the search prompt would accomplish this.
That's true, and you'll get the output in fields.awk's normal record-by-record display, including all fields in all records.

But what now if you want to create a list of just two or three fields of each record, without tedious editing?

For example, Suppose you have made a full printout of a database with field structure
FIELD1 || FIELD2 || FIELD3 || FIELD4 || FIELD5 || FIELD6 ||

and fields.awk creates a full-record printout of “n

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#80 Post by Pete »

Hi cabbie0

That is real handy, gonna give it a try.
Thanks for sharing.

Pelo

Databases ... for pleasure...

#81 Post by Pelo »

"My view is that Windows users may be even more reluctant than Linux ones when it comes to using the CLI."
First CLI needs to speak english, what can eject 95% of people having a computer.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Scripting LANGUAGE

#82 Post by cabbie0 »

Pelo wrote:
First CLI needs to speak english, what can eject 95% of people having a computer.
I hadn't thought of that, very true.

@musher0
BONNE FÊTE DU CANADA!

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#83 Post by Pete »

@Pelo

I do agree with you that the CLI can be rather cryptic at times.
One way to make it more "English" would be to use Aliases.

In general I don't like aliases as it makes your script less portable but
if it helps in the beginning then go for it.
You can always still use the original command as well.

You could even make it more "French" if you like. :)

For example if you type this:

alias imprimer='echo'

Then you can use imprimer instead of echo, like so:

imprimer "Bonjour Monsieur Pelo"

Just don't use reserved words and to make the aliases persistent across reboots, place them all in ~/.bashrc or ~/.bash_aliases depending on your setup.

Pelo

If English speaking people want to use a Database;

#84 Post by Pelo »

it was just a remark. If English speaking people want to use a Database; it's a good idea.
As i Told somewhere, Puppy package manager is a database (data are stored in hidden .packages under root. You can use these data to test fields.awk.
separator | can easily be replaced by comma or semicolon

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#85 Post by Pete »

@Pelo

OK, understood.

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#86 Post by Pete »

@cabbie0

I got my Android toy and downloaded and installed ZShaolin.
Works great and I like the feature that it has 4 terminals, but how does one enter
CNTL+C?
Sometimes it gets stuck and I need to terminate a script.
Tried volume down + c but no go.
Any ideas?

T.I.A.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Zshaolin Special Keys Mapping

#87 Post by cabbie0 »

@Pete

Hi,
Glad to hear you've managed to get Zshaolin installed on an Android device.
I use the Volume_Up button for CTRL (and Volume_Down for ESC).
But that is because I specifically mapped them to those keys.
In the startup screen for Zshaolin, you should see a 6-part menu:
[Start ZShaolin] [Install System] [Shutdown] [Keyboard] [Options] [Website]
Select [Options] (I know, you'd think it would be Keyboard....)
At the bottom of [Options], see [SPECIAL KEYS MAPPING]
There is has special options for both the CTRL and ESC keys you can select.
In each case the opions (on my version) are: Jog ball, @key, Left Alt, Right Alt, Vol Up, Vol Down, Camera Key, or None.

Hope that helps.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Read Search Output file processing, continued.

#88 Post by cabbie0 »

@all

Follow up to previous post on filtering "Read Search Output" files for displaying 2 or 3 selected Fields only.

Although the "egrep" command sequence works, it is a bit too complex to be practical
unless you could make a "one-liner" script which takes your selected fields in the form of command-line arguments and inserts them into the "egrep" sequence, so you don't have to remember the syntax every time.

I tried this, but found that transferring command line arguments as variables into such a script is VERY problematic.
The syntax section " '[SELECTION1]|[SELECTION2]' "
is the sticking point as the single quotes on either side render the variables ($SELECT1 etc) as meaningless text and if you escape the single quote with "\" then the syntax itself is rendered inoperable. -- "Catch22"

So instead I decided to use AWK for both sections of the script; that is, the the multiple-argument section and the blank line insertion section.
Two separate "one-liner" filter scripts called OPselect2 and OPselect3 for filtering out 2 or 3 FIELDS respectively.
The active code for each as follows:

Note each code string is two separate awk scripts; the output from the first (highlighted red) being piped as input to the second. Consequently the entire code string must be on a single line and not wrapped into two separate lines.
The first parts (in red) select only lines containing the FIELD names you enter on the command line.
The second part (in black) is the same for each script since it simply inserts a blank line in front of the first selection, for easier readability.
The command sequence is simple to remember:
OPselect2 [Field_selection1] [Field_selection2] [filename] (where filename is your output file generated by fields.awk)

EXAMPLE: Let's say we have a short output file from a database of tree types. Filename=treesout.txt
==> Matching record 1 :
COMMON NAME = Maple
LATIN NAME = Acer
FOLIAGE TYPE = deciduous
*******************************************

==> Matching record 2 :
COMMON NAME = Oak
LATIN NAME = Quercus
FOLIAGE TYPE = deciduous
*******************************************

==> Matching record 3 :
COMMON NAME = Pine
LATIN NAME = Pinus
FOLIAGE TYPE = evergreen
*******************************************

==> Matching record 4 :
COMMON NAME = Willow
LATIN NAME = Salix
FOLIAGE TYPE = deciduous
*******************************************

==> Matching record 5 :
COMMON NAME = Yew
LATIN NAME = Taxis
FOLIAGE TYPE = evergreen
*******************************************

This is a special case where the FIELD names are actually two separate words. The command line arguments can only be one-word, but the script only needs a recognizable tag so here we can just enter the first word from each FIELD as arguments:
OPselect2 COMMON LATIN treesout.txt

and the output should be:

COMMON NAME = Maple
LATIN NAME = Acer

COMMON NAME = Oak
LATIN NAME = Quercus

COMMON NAME = Pine
LATIN NAME = Pinus

COMMON NAME = Willow
LATIN NAME = Salix

COMMON NAME = Yew
LATIN NAME = Taxis

------------------------------------
Attaching a copy of each filter script for those who would like to try them.
Note these are FAKE zip files. Just rename them OPselect2 and Opselect3.

cheers
Attachments
Opselect3.zip
Filter script for selecting THREE fields from a fields.awk Read/Search output file.
Syntax = OPselect3 [Field_selection1] [Field_selection2] [Field_selection3] [filename]
(758 Bytes) Downloaded 190 times
Opselect2.zip
Filter script for selecting TWO fields from a fields.awk Read/Search output file.
Syntax = OPselect2 [Field_selection1] [Field_selection2] [filename]
(664 Bytes) Downloaded 191 times
OPselect_code.jpg
One-liner awk scripts for filtering Read-Search output files.
(72.63 KiB) Downloaded 233 times

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

Re: Zshaolin Special Keys Mapping

#89 Post by Pete »

cabbie0 wrote:@Pete
....
There is has special options for both the CTRL and ESC keys you can select.
In each case the opions (on my version) are: Jog ball, @key, Left Alt, Right Alt, Vol Up, Vol Down, Camera Key, or None.

Hope that helps.
Perfect, thank you.
I thought that the vol up/dn + letter key combo was an automatic Android thing.
I'm such an Android noob. :lol:
Last edited by Pete on Sun 03 Jul 2016, 09:43, edited 1 time in total.

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#90 Post by Pete »

@cabbie0

Nice addition those two scripts.
A thought, since the syntax after the pipe command is identical for both one-liners, perhaps including a bit of code to detect how many arguments the user supplied and the script would automatically branch to either 2 or 3 fields.
The number of arguments available in $#.

Another idea, I'm thinking of creating a sort of launch script from where one could start fields.awk or addcols or your latest addition.
So this launch script could be a main menu with options something like:

(S) Start fields.awk
(A) Add columns
(E) Extended search - 2 or 3 fields
:
:
:
:
(X) Exit

This way we can add new functions/one-liners easily.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Extensions

#91 Post by cabbie0 »

@Pete
Hi,

Yes I thought about using $# to automatically detect the number of fields, and it could work.

But there is one thing to consider. As is, OPselect2/3 will output to the screen.
If you want to save the output to a file, you'd be adding something like "> [savefilename]" at the end, which would be adding to the command-line argument total.
Could be done, but you'd have to decide beforehand whether the save_output function is default in order to get the argument numbering correct.

As for the other "launch script" idea, yes cool.
I gather the extended features (addcol etc) would automatically be acting upon output files in some common directory?
Mind, filtering can be done on either search-output files as with OPselect, OR it can also be done on the raw database files in DB_FILES storage directory.
That is usually a better option for "addcol" because it's adding the entries in whatever field contains some numerical data.
And you can combine it with a "grep" command to add the numerical data column for only those records that your grep filter defines.

OPselect type filters could be added to the "READ" function of fields.awk, but it would be at the cost of additional prompting every time you wanted to save a search output file...
Hmm.

ttyl

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#92 Post by Pete »

Hi cabbie0

What I was thinking is using a temp file similar to what you use for searches.
As regards directories, fields.awk stores which directory is used for the DB files, so could use a similar arrangement for where the search files are created.
At the moment they default to ~/Desktop but could also have an option to change it via a fields.awk setup menu option.

It is a bit more work in the beginning but once it's written, it will allow for easy inclusion of more features to fields.awk by way of menu selections which would invoke different scripts or "modules".

Your thoughts?

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Output filtering module idea

#93 Post by cabbie0 »

@Pete

Hi,
Yes, that is a possibility alright.
At the end of the read_from function where printing of the search output occurs:

case $PRINTORNOT in
[Yy]*) printf "`cat $OP`\n" | fold -s >> ~/Desktop/$SAVEAS.txt
echo "Your file was saved as $SAVEAS.txt in ~/Desktop.";;

Instead of immediately ending the "case" module with "esac" and returning to "main", you could clear the screen before the "echo" line and then issue a prompt:
echo "Filter output file? "; read REPLY

Only if the answer was Y would further prompts appear for selection of type of filter (addcol, OPselect, etc) and their appropriate sub-prompts and branch codes.
Otherwise the code would return to the main screen as usual.

You'd have to remember to save the "esac" module end (and the double semicolon ;; oops that would require care) for the very end of the submenus...
I won't be doing this immediately ;)

User avatar
Pete
Posts: 660
Joined: Sun 02 Mar 2014, 18:36

#94 Post by Pete »

Thanks for the tips cabbie0.
I will see what I can can up with during my spare time.

cabbie0
Posts: 26
Joined: Thu 23 Jun 2016, 05:19
Location: British Columbia, Canada

Enhanced Fields.awk

#95 Post by cabbie0 »

Hello all,

@Pete
You got me thinking about some good additional functions, and before I knew it I was coding again...

I have added some new functionality to fields.awk.
Feel free to download and tinker with the upgraded version, fields_ext.awk.
This is a real zip archive (.bz2) so you have to unzip it.

The two main additions are:
1) a column-adding feature (function "addcol") added to the main menu (select T for tally/total)
2) a search output filtering option, offered whenever you save a read-search to file.

Also, though the default search-output file folder is still ~/Desktop, you can now easily change that
by editing the line (found at the script beginning just after the preamble):
SEARCH_DIR=~/Desktop
to reflect your desired location. Just make sure it actually exists ;)

A handy short database file (weather_data.txt) for testing the "tally" function:
@@@DATE::mm_RAIN::uV_rads::lightning_strikes::
05-10-16::60::0.89::3::
05-15-16::12::0.54::1::
05-20-16::74::2.1::6::
05-25-16::98::0.34::3::
05-30-16::987::.87::7::
06-05-16::0::.23::0::
06-10-16::45::0::14::
06-15-16::10::.59::1::
06-20-16::10::1.0::2::
06-25-16::36::.10::15::

Select this file and choose "T" from main menu, then select field#2 for adding:
Note it offers an option to filter the tally, that is add the numbers in selected field only for records pertaining to a search string.
In this example I've selected for only the "June" entries by using the search string "06-"

DATE || mm_RAIN || uV_rads || lightning_strikes ||

Which field column would you like to total? (Enter an integer)
2
Total of column#2 = 1332.00
___________________________________________
Try a filtered tally?
y
Enter filter string:
(Default case-insensitive; only those records including the string will be tallied)
06-
The following is your _filtered_ tally result:

Total of column#2 = 101.00
___________________________________________
Hit ENTER to return to main menu ...

==============================================================

For the search output filter function using the same small database, if you save a search file including all the records, then select a filtered output including only DATE and mm_RAIN, the interface looks like this:

Your file was saved as weather.txt in /home/iforsyth/DB_FILES/SEARCHES.
Filter output?
y
DATE || mm_RAIN || uV_rads || lightning_strikes ||

Select # of fields to display (2-4)
2
Enter string for field selection#1:
DATE
Enter string for field selection#2:
mm_RAIN
Your filtered output was saved as weather_f2.txt in /home/iforsyth/DB_FILES/SEARCHES.
Hit ENTER to return to main menu ...


Note: the output filter can handle either 2, 3, or 4 field selections for filtering.
Hope you find these additions useful.
Cheers,
Ian
Attachments
fields_ext.awk.bz2
Enhanced fields.awk, includes "addcol" function plus output filtering.
(4.72 KiB) Downloaded 210 times

Post Reply