Need on-screen keyboard with mouse functions

A home for all kinds of Puppy related projects
Post Reply
Message
Author
User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#16 Post by greengeek »

Thats great - it would be useful to have someone familiar with xdotool to offer some advice.

I just had a tinker with xvkbd - it's quite good. I think it could be very useful, especially if it was also possible to write a program / script that is capable of moving the mouse pointer across the entire screen in a row by row scanning motion (over the top of any/all open windows).

If the mouse pointer could be made to scroll slowly across the top line of the screen, then step down 5mm and scroll back, then step down 5mm and scroll across again etc etc it would eventually cover the entire screen and allow a mouseclick to access all functions that xvkbd (or it's focused window) was presenting. (Very slow of course, but I'm just looking for a "proof of concept" idea at the moment. Something might be better than nothing...)

EDIT: xdotool does not seem to respond to WriteAid character depressions, but does respond to xvkbd character depressions. So, if a way could be found to automatically scroll through the xvkbd keys onscreen, it would be possible to program some of them to do special functions like File, save, file open, move mouse etc etc.

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

Delay

#17 Post by standard7452 »

Sorry for the delay in responding....I was dumb and didn't subscribe to my own thread, and I have 2 8 month old twins who have had the stomach bug.

The scanning function of gok/onboard allow the user to use a single action input to start "scanning" and logically determine which column, row, button they need.

So imagine the following....a person who has no control of arms/legs, and only a single range of motion in their neck. What we do now is put a spst momentary switch, let's say under their chin.

Currently, the only way to make the switch work now (in windows) is to buy a usb interface with a driver to recognize the switch as an input device.

What I did, was disassembled the keyboard to find the ic that runs let's say the pause/break key...a key that I doubt would come into conflict with normal internet surfing or email and connect the switch across the appropriate terminals to make essentially a "one button Keyboard"

Currently the interfaces are like $200, so for the price of a junk keyboard the solution works.

Now, I need that key to start the scanning process in something like GOK or Onboard.

The fundamental problem I found with GOK is that the original developers assigned all of the scanning functionality to any one of the mouse buttons.

This creates a conflict with the pointer, because every time you click to make it scan it also clicks wherever the pointer is.

Onboard appears to let you select another keyboard key as the scanning activator, but does not have mouse controls.

Its hard to imagine but I could post a video of someone using a very expensive paid application to show what I am talking about.

Thank you for your response, and sorry for the delay.

Will

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

scanning

#18 Post by greengeek »

standard7452 wrote:Now, I need that key to start the scanning process in something like GOK or Onboard.
Would it be problematic if the scanning started automatically as soon as the PC booted?

Also - does this system have to be designed so that the single-switch user is the only person available to start the required processes, or is it likely that there will be another person available to boot and commence the necessary programmes?

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

#19 Post by standard7452 »

Everyone thank you for your help.

I appreciate your time and patience while I try to explain this right because you guys are so close to solving this problem.

Its just so difficult to fully explain in one shot.

Onboard Onscreen Keyboard provides an option so that a keyboard key can be used to activate a scan of the keyboard layout. Here is a list of what happens for this:

1. The user presses a predefined keyboard button on the physical keyboard once and lets off.
2. The entire first row of the onscreen keyboard is selected so lets say the number row (` through =), meaning that if the user wants a key in that row they need to press the predefined button a second time.
3. From there, each key is selected from left to right. The user must press the predefined key again to make their final selection.
4. The process starts over for the next character or function the user needs.

With Onboard this part already exists, but there is no way to use the mouse.

Now think of "Mouse Keys" for windows or the use numberpad to control the mouse in Ubuntu.

if you hold the 8, the mouse cursor moves up relative to its position, 4 goes left, 6 goes right and 2 goes down.

if I could add six buttons to any onscreen keyboard that has the scanning feature mentioned above that ran a script for xdotool to move the mouse in a direction until the user hits the predefined physical keyboard button again. and also have a button for right, left and doubleclicks that would be it.

so like buttons on xvkbd that ran something like xdotool mousemove 0,5 on a loop until the user hit the predefined keyboard button.

hope this makes sense, if not I can make a video to explain it.

sorry and thanks!

Will Best

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

#20 Post by greengeek »

standard7452 wrote:if I could add six buttons to any onscreen keyboard that has the scanning feature mentioned above that ran a script for xdotool to move the mouse in a direction until the user hits the predefined physical keyboard button again. and also have a button for right, left and doubleclicks that would be it.
So is this similar to what you mean?

Programmatically assign the following keys with the attached functions as shown:
F4 = "slowly scroll left till user hits switch"
F5 = "slowly scroll up till user hits switch"
F6 = "slowly scroll down till user hits switch"
F7 = "slowly scroll right till user hits switch"

User avatar
Ted Dog
Posts: 3965
Joined: Wed 14 Sep 2005, 02:35
Location: Heart of Texas

#21 Post by Ted Dog »

Ah, you got around the expensive switch, I'd like to see a video setup.

Also do you know if you keep a lazer mouse still and just move the pad its the same and moving the mouse, would a upside down mouse and a padded semi ridged chin strip made of mousepad work. Also the mouse buttons are easier to reuse over a keyboard.
Just a thought.

I asked my local chapter of the MDA if there was any interest in a project like this... She was not expecting this type of topic so she will get back with me in a few weeks, after the Shamrock for MDA fund drive is over. :wink:

While on the topic, MDA fund drive is underway, please donate,
MDA saved my life, I was going down hill with a form of MD that is very rare. Normal doctors where unable to classify or treat my condition for 12 yrs. When I visited the MDA clinic they where able to id the problem and I had a workable prescription before I left, also since it was rare they had me stay the rest of the day and be seen by everybody in the department and a few students.
If I was seen by that many doctors and had all the expensive tests run it would have easily been 10,000 dollars, my cost $86 rest covered my MDA.

starhawk
Posts: 4906
Joined: Mon 22 Nov 2010, 06:04
Location: Everybody knows this is nowhere...

#22 Post by starhawk »

A thought that occurs to me --

Could these people manipulate a directional-pad sort of setup, like on a Game Boy? If so, a special switch (not expensive!), an Arduino Leonardo chip (the whole board is not necessary), and a PS/2 cable is all you'd need.

But they'd need to be able to set their chin on the switch actuator and then move the chin slightly to make the circuit do its thing. They'd also need to be able to move to a second and third switch for left- and right-click.

Directional switch like this --> http://www.sparkfun.com/products/10063
Other two switches as needed, maybe like this? --> http://www.sparkfun.com/products/9337
Arduino chip --> http://www.sparkfun.com/products/10524
28pin DIP socket (for the chip) --> http://www.sparkfun.com/products/7942
...as a note, that is an utterly atrocious price. Mouser Electronics sells those for less than $0.25! (But the added shipping cost isn't worth it.)

The Arduino would need to be programmed, using the ps2dev library here as a starting point --> http://playground.arduino.cc/componentLib/Ps2mouse

A further note -- the switch I mention is going to be a real doozy to solder, simply because it's surface-mount. Probably the best way to deal with this would be to use this "breakout board" that puts things into through-hole friendly form. You would not need to buy the switch separately -- the "breakout" board includes everything needed here. There is an additional "select" output on that switch which would not be used -- that can be implemented in other buttons. (This is likely to be far, far less frustrating than "press the button and push, but don't push too hard or you'll click on something!")

So, per device, you'd be looking at a cost of $12.50 plus shipping, labor, and whatever sort of PCB you'd want it mounted on -- Radio Shack actually has a pretty nifty "perfboard" selection (perfboard is a PCB with plated holes spaced at 0.1" intervals, primarily for prototyping and for hobbyists who can't afford custom PCBs)... I'd recommend their Catalog # 276-158 board, at about 2.8"x3.7". See here --> http://www.radioshack.com/product/index ... Id=2103798

I daresay that, given some experimentation and such, I could quite likely build one of these myself as a prototype. I don't think you'd want me building a lot of them for you (I've got motivation issues, among other things) but I'd be happy to assemble a prototype if I can scrape together the money...

starhawk
Posts: 4906
Joined: Mon 22 Nov 2010, 06:04
Location: Everybody knows this is nowhere...

#23 Post by starhawk »

I have a much better design for this. Let me know if you're interested, and if so, I'll send you the plans.

Total cost should be under $30 including all parts and shipping. (Labor is extra.) It is 100% workable, with all off-the-shelf components, and can be assembled by an experienced maker in probably about an hour. (That means it'll take me two hours at least!)

I still have to write the programming to make it work -- but it WILL work!

What's more -- if the system this is used on has an integrated keyboard (i.e. is a laptop), there is no need for any other input device so long as an onscreen keyboard is readily available! That is -- this can REPLACE the single-button keyboards entirely.

For systems that will not boot up without a keyboard present -- simply find a PS/2 keyboard, open it up, and pull out the electronics inside. You only need the PCB and PS/2 cable to remain intact -- discard the rest. Wrap it up tight with electrical tape, hook it up, and stash it somewhere. Presto! One keyboard that will never experience another keypress, but fool the computer into thinking all is well. Simple and brutal, but remarkably effective.

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

#24 Post by standard7452 »

greengeek wrote:
standard7452 wrote:if I could add six buttons to any onscreen keyboard that has the scanning feature mentioned above that ran a script for xdotool to move the mouse in a direction until the user hits the predefined physical keyboard button again. and also have a button for right, left and doubleclicks that would be it.
So is this similar to what you mean?

Programmatically assign the following keys with the attached functions as shown:
F4 = "slowly scroll left till user hits switch"
F5 = "slowly scroll up till user hits switch"
F6 = "slowly scroll down till user hits switch"
F7 = "slowly scroll right till user hits switch"
Yes! But, the keyboard has to be "scannable" as well so that the person can first select the f4 key using row, column deduction, and then the mouse would start, and then the mouse would stop when they hit the switch again.

Functionally, this is exactly what it would be with 3 other keys, left click, right click, and double click.

When I am back at work tomorrow (I got the stomach bug my kids got), I will take a video of someone using a super expensive solution just so you guys see what the end functionality is.

Thank you again for your help!

I will post the video tomorrow which I think will clear up the details of what I am looking to do (obviously with much help, ty)

Will

User avatar
greengeek
Posts: 5789
Joined: Tue 20 Jul 2010, 09:34
Location: Republic of Novo Zelande

Re: Delay

#25 Post by greengeek »

standard7452 wrote:The fundamental problem I found with GOK is that the original developers assigned all of the scanning functionality to any one of the mouse buttons. This creates a conflict with the pointer, because every time you click to make it scan it also clicks wherever the pointer is.
I would just like to clarify this if possible - is the conflict caused because the pointer MOVES when you click the mouse button (because of the mechanical pressure on the mouse), or do you mean that the conflict is that you cannot trigger the scanning unless the mousepointer is in one correct place only? (and you want the mouse to be elsewhere at the moment you trigger scanning...)

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

Re: Delay

#26 Post by standard7452 »

greengeek wrote:
standard7452 wrote:The fundamental problem I found with GOK is that the original developers assigned all of the scanning functionality to any one of the mouse buttons. This creates a conflict with the pointer, because every time you click to make it scan it also clicks wherever the pointer is.
I would just like to clarify this if possible - is the conflict caused because the pointer MOVES when you click the mouse button (because of the mechanical pressure on the mouse), or do you mean that the conflict is that you cannot trigger the scanning unless the mousepointer is in one correct place only? (and you want the mouse to be elsewhere at the moment you trigger scanning...)
http://m.youtube.com/#/watch?v=Mn-HLmEP ... n-HLmEPI_s

This is a video of what I mean by "scanning" that uses the win7 OSK.

In the first part he uses the "Point and Click" method where he selects the character or key he wishes to use with the mouse pointer. In the second part he uses the "Scanning" mode to allow him to press a predefined button like "space" to get a row selected, and then three characters, then the final character the user wants.

GOK has a keyboard layout that included mouse functionality like you we're talking about with xdotool. There is buttons (like enter, space, letters, buttons), except they are for mouse up (xdotool mousemove_realtive 0,5 on some sort of loop) and mouse down, etc. these buttons allow for the user to access both the keyboard and mouse using a single switch and the deductive logic part.

GOK's conflict lies in the fact that they used a mouse button exclusively to initiate this scanning process. So if you want to move the mouse using the OSK with GOK, every time you want to change direction or open a context menu or whatever, the core pointer itself clicks.

[/url]

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

#27 Post by standard7452 »

starhawk wrote:A thought that occurs to me --

Could these people manipulate a directional-pad sort of setup, like on a Game Boy? If so, a special switch (not expensive!), an Arduino Leonardo chip (the whole board is not necessary), and a PS/2 cable is all you'd need.

But they'd need to be able to set their chin on the switch actuator and then move the chin slightly to make the circuit do its thing. They'd also need to be able to move to a second and third switch for left- and right-click.

Directional switch like this --> http://www.sparkfun.com/products/10063
Other two switches as needed, maybe like this? --> http://www.sparkfun.com/products/9337
Arduino chip --> http://www.sparkfun.com/products/10524
28pin DIP socket (for the chip) --> http://www.sparkfun.com/products/7942
...as a note, that is an utterly atrocious price. Mouser Electronics sells those for less than $0.25! (But the added shipping cost isn't worth it.)

The Arduino would need to be programmed, using the ps2dev library here as a starting point --> http://playground.arduino.cc/componentLib/Ps2mouse

A further note -- the switch I mention is going to be a real doozy to solder, simply because it's surface-mount. Probably the best way to deal with this would be to use this "breakout board" that puts things into through-hole friendly form. You would not need to buy the switch separately -- the "breakout" board includes everything needed here. There is an additional "select" output on that switch which would not be used -- that can be implemented in other buttons. (This is likely to be far, far less frustrating than "press the button and push, but don't push too hard or you'll click on something!")

So, per device, you'd be looking at a cost of $12.50 plus shipping, labor, and whatever sort of PCB you'd want it mounted on -- Radio Shack actually has a pretty nifty "perfboard" selection (perfboard is a PCB with plated holes spaced at 0.1" intervals, primarily for prototyping and for hobbyists who can't afford custom PCBs)... I'd recommend their Catalog # 276-158 board, at about 2.8"x3.7". See here --> http://www.radioshack.com/product/index ... Id=2103798

I daresay that, given some experimentation and such, I could quite likely build one of these myself as a prototype. I don't think you'd want me building a lot of them for you (I've got motivation issues, among other things) but I'd be happy to assemble a prototype if I can scrape together the money...
This sounds interesting! I need to take a video of how they access so that you could look...as far as prototypes I would fund it, but first take a look to at the vid to see if it is applicable....I should have it n tomorrow,

starhawk
Posts: 4906
Joined: Mon 22 Nov 2010, 06:04
Location: Everybody knows this is nowhere...

#28 Post by starhawk »

I eagerly await your video... ;)

Minus the programming part (which will be hard for me, but I think I can do it) the entire design is basically done. It uses parts from Mouser Electronics (www.mouser.com) and Radio Shack. There will be one part that must be scrounged (the PS/2 cable) but that's not a terribly big deal IMO. Nearly anyone should be able to build it in an afternoon -- so it doesn't need factories and mass-production, it can be built by hobbyists and makers the world over! (This also lowers the cost of things dramatically.)

I have a couple of things I'd want to talk to you about, to get the design spot-on, but at this point I'm just doing fine-tuning.

The problem with the programming is that I'm used to a very old programming language that the world has largely moved on from -- QuickBASIC -- and the Arduino uses a subset of C/C++, which is completely different. I'll have some learning to do!

I would like to open-source this design if at all possible... I'll be glad to assemble one or two for you, and we can figure out how to tweak them, but as I said above -- if you can hold a drill, and if you can solder, then you can build this thing yourself.

I will also mention -- if you need to use a USB-to-PS/2 adapter with this, for computers that do not have PS/2 ports, you need to steer clear of these two styles of adapter --
Passive, wires-only adapters, like this one.
Active but super-cheap adapters like this one.

The one in the first link won't work, and it could damage things. Those rely on the mouse being able to detect what kind of port it's attached to, and switch by itself. This device can't do that!

The one in the second link will "sort of" work -- those have HORRIBLE filtering (if any at all) in them -- I had one and it was so jittery that I just got rid of it. The mouse cursor dances a jig even when the mouse ISN'T moving, and when it is moving the dancing gets worse. By the way, these come in both beige and black -- so don't be fooled, lol.

I can't speak to how well the following adapters work, but money provided I'll gladly test them --
http://www.ebay.com/itm/280775192717
http://www.ebay.com/itm/111011014372

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

#29 Post by standard7452 »

I made a couple of videos to explain what the project entails...


...I apologize if I seem sassy on the video, I had to do it in the early morning, and I am not a morning person.

I just went over the basics, not because I think anyone is dumb, it just is such a weird thing to explain to folks....

Plus this is the first time I have ever made a vid like this so I didn't want to look too stupid....

Thanks for your help!

part one - http://www.youtube.com/watch?v=BaKmdhGexVk

part two - http://www.youtube.com/watch?v=6b857d5G2Qs

User avatar
Ted Dog
Posts: 3965
Joined: Wed 14 Sep 2005, 02:35
Location: Heart of Texas

#30 Post by Ted Dog »

Thanks, DL them now, internet is slow for me.... :(

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

Rethinking the Keyboard for single-switch

#31 Post by mikeslr »

Hi All,
The following was written before I watched Will's video. But I think most of it holds up. So I'm submitting it "as is" or "was."
The discussion now appears to be entering into a technical area beyond my comfort zone. Except for checking in once and awhile, and seeing if I can recruit others having needed expertise, this may be my last post on this thread.
Although I have always been driven curiosity about everything, shortly after WWII, when I was about 6 and TVs had yet to become mainstream, I took apart the family's radio. No one knew how to fix it and its replacement cost about a week's savings. To say the discussion which followed was not an encouragement of my tinkering with appliances would be euphemistic.
Now that I better understand the problem, at present I have only one thing to add. It will probably complicate things.
We are creatures of habit. Much --perhaps most of what we do, or at least how we do things-- is the result of our internalization of our culture's adaption to its environment, including the existent technology. We learn it, do it, it becomes the way we do it and, thereafter, we don't think about it. Having habits is efficient as anyone will attest who has had to wait while someone else tries to locate keys or eye-glasses “just left someplace.
Attachments
Keyboard.jpg
Possible Keyboard Arrangement
(59.23 KiB) Downloaded 644 times

starhawk
Posts: 4906
Joined: Mon 22 Nov 2010, 06:04
Location: Everybody knows this is nowhere...

#32 Post by starhawk »

One more question before I give up on my hardware solution.

Can these people reliably tilt their chin forwards, backwards, left, and right, about a half-inch, without problems?

If this gets into pure software I'll have to bow out. I can probably learn arduino code, but even a linux shell script is likely beyond me right now (although I'll have a look if it becomes necessary).

EDIT: even tilting the chin 1/4 or 3/8 inch, would be enough... just need to be able to do it enough to bump into something is all.

standard7452
Posts: 20
Joined: Mon 25 Feb 2013, 23:11

#33 Post by standard7452 »

starhawk wrote:One more question before I give up on my hardware solution.

Can these people reliably tilt their chin forwards, backwards, left, and right, about a half-inch, without problems?

If this gets into pure software I'll have to bow out. I can probably learn arduino code, but even a linux shell script is likely beyond me right now (although I'll have a look if it becomes necessary).
Starhawk, I think your hardware solution sounds awesome so I don't want you to think your efforts were in vain for a second.

It's just that the guys who get the shaft on computer access only have a single motion (that isn't even always predictable). So the only way to delineate it is by steps--move mouse, click mouse, wait for A, etc.

Imagine putting a 10kg-20kg weight around your neck and you could only interact if you could put the part in the right place....and you are disabled.

Thank you so much for your help....no good deed goes unchecked.

starhawk
Posts: 4906
Joined: Mon 22 Nov 2010, 06:04
Location: Everybody knows this is nowhere...

#34 Post by starhawk »

OK :oops:

I'm sure someone will help you with a shell script, then. Not something I can really do.

User avatar
mikeslr
Posts: 3890
Joined: Mon 16 Jun 2008, 21:20
Location: 500 seconds from Sol

If the mouse offends thee, pluck it out

#35 Post by mikeslr »

Hi standard7452 and all,

Seems I was mistaken about having little more to add.

I previously indicated that I was able to obtain a functioning version of onboard in jejy69's lxpup-aptget-test by using its included synaptic to install the necessary debs. With onboard onscreen and geany opened clicking the onscreen keys sent the keys selected to geany which displayed them. I'm not certain, however, if onboard is entirely functional under lxpup-aptget. I called up its config module and turned on scanning. Thereafter, various sections of the keyboard were highlighted, but (1) selecting mouse as the keyboard device and checking "mouse only" froze the application except to allow pressing the Enter key to uncheck it and (2) after unchecking "mouse-only" clicking a key on the onscreen keyboard while in the scan mode no longer sent key-strokes to geany. Turning off that mode restored onboard’s communication with geany. Similarly, unless scanning was turned off, onboard did not communicate with either opera or firefox. Hopefully, that problem is simply because I don't really know how to configure onboard or can be overcome. Exploring onboard has overcome my concern that an on-screen keyboard would occasionally obscure other on-screen windows: its display can be dragged and resized so that it uses only space immediately above the taskbar. Although other windows may extend under that display, there's probably a workaround. Interestingly, when set to “grid
Attachments
WP51Dos.jpg
Wordperfect5.1 for dos running under dosemu
(124.23 KiB) Downloaded 659 times

Post Reply