Goals:
The goal of my project is to rehabilitate patients with hand injury due to stroke in a fun effective manner.
Through the use of my VR Piano I plan to improve the motion of fingers through repeated exercises. The user will follow a ball corresponding to the correct key on the piano. One benefit of using my VR Piano is that it can be customized to focus on individual finger (fractionation) or the whole hand. Also the range and speed can be adjusted accordingly to the patients needs. For example a song that can be designed to maximize duration of finger bending by strategically positioning the keys to be further apart causing the user to hold a position for a longer period of time. This could also be achieved by changing the speed at which the glove moves across the screen increasing the time that the user has to hold a position. One inclination is that the strengthening has a limitation because there is no force feedback (but it could be added with the use of the Rutgers Glove).
All the code, state machine design, and objects used in my project are my own work and have been created from scratch. In this project I used the concept of hierarchy, state machine design, and plenty of route statements. The nodes that I used were the AudioClip node, Script node, Text node, ImageTexture node, Viewpoint node, and the IndexedFaceSet node.
Figure 1.1
Figure 1.2 User can choose a different predefined viewpoint by clicking on the "View" button in the web browser.
Figure 2.1
Figure 2.1 represents the user turned the keyboard ON and pressed the correct key, which displays "Good, now press the next key." This is also good because if the next key is the same key the user will know to press the key again.
Figure 2.2
Figure 2.2 represents the user pressing the wrong key and with an encouraging message to "Try again."
Problems I was faced with:
The two biggest problems that I was faced with were the fact that collision detection was not available for us to use and to involve the other fingers with the rehabilitation exercise. Two smaller problems that I encountered were how to implement the songs and how to perform an operation on a constantly updated value.
My whole project was based on the interaction/collision of the glove’s fingers and the keys on the VR Piano. The way that I went about solving this problem was to determine the gloves position on the screen and if the user was bending the correct finger over the correct key. These are the conditions that need to be satisfied in order for the sound to be played.
Example pseudocode to solve collision detection:
If (user is bending finger at the correct angle && if the finger is on top of the key && if correct key to be pressed) then play sound corresponding to that key.
By using gesture recognition techniques I was able to involve not only the index finger but also the thumb and pinky finger. When the thumb is bent the glove will move to the left and while the pinky is bent the glove will move to the right. The position that the glove is in is stored in a variable called NewMove. This variable is used to test if the glove is over the correct key.
Example pseudopodia used in pinky gesture recognition:
if (NewMove < max length of keyboard) then increment the x position of the glove
Example pseudocode used in thumb gesture recognition:
if (NewMove > min length of keyboard) then decrement the x position of the glove
One of the smaller problems that I was faced with is how to implement the songs that the user would play and how would they know what key to press. I solved this by using a single array of values that range from 1 to 4, each one representing a key on the VR Piano. With this flexibility the doctor can change the song to allow the user to follow a predefined song that can be designed to maximize duration of finger bending. For example if the array is array[1 4 1 4 ….] then the patient will have to hold down his/her pinky and thumb to reach the far ends of the piano. A ball was placed over the correct key that the user was to press. This was accomplished by using the song array to determine which key the ball was to be placed on. Each key has a known position so I used switch( key that the ball was to be on)
Example pseudocode to solve learning ball placement:
Switch( key where ball needs to be placed){
case key 1 :
x position of key 1 = 3;
y position of key 1 = 8;
break;
.
.
}
The other small problem that I was faced with was the fact that the value of the finger was constantly changing. In order to solve this I used Semaphores to lock the value, perform the correct operation, and then grab the next value. The lock was based off a specific value and would only open if it was at the specific value.
Example pseudocode for semaphore used in program:
If ( lock == open ) enter critical section and perform the task
else wait for lock to open.
State Machine Diagram:
The following pseudo code is based off the state machine design for the Virtual Reality Piano.
Inside the function Index() you will find the actual code developed from this pseudo code:
Function Index(){
Lock = closed
cur_state = next_state
switch(cur_state){
idle state:
wait for lock to open
if finger is above key next_state = key pressed
and close lock.
--------------------------------------------------------------------
key pressed state:
if key 1 is keypressed = key1
if key 2 is keypressed = key2
if key 3 is keypressed = key3
if key 4 is keypressed = key4
next_state = correct key pressed
---------------------------------------------------------------------
correct key pressed state:
if keypressed = array[array of keys] then next_state = display good
else next_state = display try again
---------------------------------------------------------------------
display try again state:
change the z coordinate of text to display try again
change the z coordinate of text to hide good job, press next key
next_state = key pressed state
---------------------------------------------------------------------
display good state:
change the z coordinate of text to hide try again
change the z coordinate of text to display good job, press next key
next_state = play sound
---------------------------------------------------------------------
play sound state:
switch(keypressed){
case 1:
startKey1Time = eventTime;
case 2:
startKey2Time = eventTime;
case 3:
startKey4Time = eventTime;
case 4:
startKey4Time = eventTime;
default:}
next_state = update
---------------------------------------------------------------------
update state:
get next beat from array
switch(beat from array){
case 1 :
put ball on top of key 1
case 2 :
put ball on top of key 2
case 3 :
put ball on top of key 3
case 4 :
put ball on top of key 4
case 5:
END OF SONG
default:
reset array counter back to zero to get first element of array.
next_state = start
}
Variables needed:
int counter for array
int keypressed
int states (i.e. start,..update)