View source

== Subject ==
A semi-professional guitar player, called Joop, has suffered from a stroke. This caused the right part of is body to be numb for the most part. He cannot play the guitar anymore. This project is all about him and starting a journey to get him back on the stage.

== Information about Joop ==
Joop is a guitarist, who has been playing in three bands. He also performed. The kind of music used to play was music in the genre of like Pink Floyd, Led Zeppelin and Van Halen, including them. He played this using a plectrum and a lot of feeling coming from his left arm in bending the notes. This is the music we strive for to bring back to the guitar for Joop. In the summer of 2017 he suffered from a stroke. The stroke caused brain damage, which in turn led to the loss of functionality of the right side of his body. Fortunately, Joop does not experience any pain.

Parts of the right side of his body that lost functionality:
<ul>
  <li>He has some difficulty with talking.</li>
  <li>He cannot use his elbow or wrist or fingers.</li>
  <li>He cannot use his knee or ankle.</li>
</ul>

Parts of the right side of his body that are limited in movement:
<ul>
  <li>He can move his right arm just from his shoulder.</li>
  <li>He can still walk with a rather stiff leg.</li>
  <li>He can still pressure his index finger and thumb against one another.</li>
</ul>

In the six month following the incident, Joop went a lot forward with his health. After the six months the improvements stagnated.

== Objective ==
The focus of this project is to make a start of making it possible for Joop to play the guitar again. This includes finding out what are the possible ways to create an interface for him, with which he could play again. These interfaces need to be natural for him and should fit within his style of playing.<br><br>
The goal for this project is to make a clear overview of the movements Joop can make. Joop should be able to use these movements for a couple of hours at a time, comfortably. These movements should also be natural and contribute to bringing his previous style to life again or make him feel that he can use his emotions in his music. This means that Joop should be able to use these movements or combination of movements to play fast songs and solos from Van Halen but also produce controlled and meaningful sounds for music like Pink Floyd. Playing fast notes is not the only challenge. Playing chords should also be possible which presents different challenging parts than playing fast. Normally only a single string is played but with chords, several strings can be chosen at the same time. Of course, I will also come up with ideas for this but I will not experiment with this.

== Approach ==
The process that is used for this project is mostly iterative. The first step is a literature review, which is done outside of the iterative loop. <br><br>
The iterative step is the test phase. For all possible ways for Joop to play the guitar, there need to be little tests. These are shown under the section tests. They will be carried out one after the other. These tests will have a conclusion on the used movement in the test, whether they are useful or not.

== Literature Review ==
In this project a foot pedal has been made and a mechanism which replaces the strumming hand. The system on the guitar strums each string of the guitar while the player frets each string[https://hackaday.io/project/26088-adaptive-guitar]. The signals from the foot pedal are send wirelessly to the unit on the guitar, which actuates the strings based on the signal.
The strings are played by knobs that go up or down over the strings controlled by actuators. These knobs are attached to the actuators by a spring. These knobs wear out in a couple of hours.
Currently, the actuators can only play the strings at one volume.

This is a paper that shares a mechanism on the neck of the guitar which replaces the fretting hand. This done by a mechanism that pushes down the strings when commanded to using a foot pedal[http://homes.create.aau.dk/dano/nime17/papers/0102/paper0102.pdf].

This is a company that makes custom instruments voor people with physical disabilities. They do this by, for example, using conductive paint. That makes for a kind of piano like playing experience[https://makezine.com/2015/06/22/developing-musical-instruments-people-physical-disabilities/].

This company has made a universal stand. This stand is also suitable for a guitar[https://www.kayjae.com/].

“Experiment with “open tunings” on the guitar – a guitar in open tuning requires less complicated fingering of chords and can be played using a slide to bar all strings on a fret to produce the notes desired.  The guitar can be played lying face up on one’s lap, with the right hand fingers picking strings, and the left hand using the slide bar to the required fret.  This may sound complicated to a non guitar player, but a guitar teacher who is familiar with open tunings and playing blues style guitar will understand this method.  It’s important to keep an open mind and have an adaptive approach to the training.  Usually where there’s a will, there’s a way.  Playing a guitar with a slide is used in many styles of music and it’s kind of a shortcut to playing sooner.
For one-armed guitarists – tune the guitar in open “g”, place face up on lap, place steel or glass slide on left pinky to bar frets.  Attempt to strum with remaining fingers – or if the other arm has any gross movement, with that arm.”[http://www.disabled-musicians.org/adaptive-gear/]

== Experiments with Joop ==
While the right side of Joop's body cannot be used for playing guitar, the rest of his body is completely fine. Therefore, other parts of Joop's body need to take over the functionality of his right hand. Some experiments need to be carried out to see what other movements are useful and natural for Joop.
<ol>
<h3><li>Foot piano</h3>

The name of the experiment says it all. A keyboard to be played with a foot is being used to see how easy it is to get used to this kind of movement. A tube around the largest toe is used to play the individual notes.

Joop played like this for two times 15 minutes a day for a week long.

<h4>Hypothesis</h4>

At the end of the week, the ability to play the piano with the foot is there but the speed is quite low.

<h4>Actual outcome</h4>

After the one week practice, Joop was able to some songs on the piano with his foot but he needed to go slow to be able to play the songs.

<h4>Conclusion</h4>

The side-wards movement in his foot is not fast enough. So, it will not be possible to have kind of keyboard, with a key for every string. Although the playing experience was very comfortable.

<h3><li>Foot pedal</h3>

In this experiment a MIDI foot pedal will be used to 'select a string'. This is done by having a different note played at different angles of the pedal. This is done by using the software Max[https://cycling74.com/products/max/] This is needed to see if the up and down wards motion of the foot has the comfort of the previous experiment and the speed that is needed to play fast notes. It is set-up so that when a certain angle is held on the pedal, one keystroke on the keyboard can generate a note. The first key on the keyboard will trigger the equivalent note of an open string. Every half step higher on the piano will add a half step to the selected string, just like a guitar. The goal is to see how reliably Joop can pick the angles of the pedal with his foot. For the pedal six ranges are used to imitate the six strings on the guitar. Every range has the same size.

<h4>Products used</h4>
<ul>
    <li>Expression pedal[https://www.bax-shop.nl/sustain-pedaal-expressiepedaal/yamaha-fc7-expressie-pedaal/product-details.html]</li>
    <li>MIDI keyboard [https://www.keymusic.com/nl/item/roland-a-49-midi-keyboard-controller-black/]</li>
</ul>

<h4>Hypothesis</h4>

Joop is able to use this pedal for a couple of hours without any strain. Although it will be getting used to, a week is enough time to play Pink Floyd songs, with an accuracy of at least 80%.

<h4>Actual outcome</h4>

TBD

<h4>Conclusion</h4>

TBD

</ol>

== Future experiments ==
During the meetings with Joop, there were always thought of new ways to control the guitar. To try out all these ideas, an experiment should be conducted. In this section all these ideas will be explained and a possible method of testing will be presented.
<h3>Idea 1</h3>
During one of the meetings with Joop, he was asked if whistling was natural to if he played a solo. This was indeed the case because the notes automatically are played in his head. The idea came up to use a microphone to record this whistling and process the notes or the on-sets of the notes he whistled and combine that with a pedal.<br>
<h4>Testing advice</h4>
This can be tested by altering the Max-program and implementing that the on-set of the note will be played by the breath or note that is whistled.
<h4>Problems</h4>
The biggest problem is the latency that is caused by the input buffer, analyzing and the output buffer. When the input buffer gets smaller the analyzer gets less accurate. This might be because the implementation I did was not done well. It can be done if the analyzer was made more effectively.
<h3>Idea 2</h3>
After one of the visits with Joop an idea was raised to extend the foot pedal experiment to also support chords. Because it was thought that chords were easy to implement, there was not any attention given to this.
<h4>Testing advice</h4>
A camera on the neck could be useful. This camera can track the finger on the fret and play the strings accordingly.
<h4>Problems</h4>
Playing a chord using the method of splitting up the selection of a string and the on-set, is easy with single strings that need to be played. However, when more than one string need to played, the problem arises that all the string that need to be played also need to be selected. Currently, there is no easy way to accomplish the selection for the strings of a chord.

== Summary of the movements ==
All the movements that show great promise from the tests or have been thought of while conducting interviews, are in this section.

== Gear and software ==
In this section will be a list of stuff that is bought for the tests with the links shops where they were bought. Not everything on this list used in the experiments but they are probably necessary for future experiments.
<ul>
    <li>The software tool Max MSP[https://cycling74.com/products/max/]</li>
    <li>Roland A49 MIDI keyboard[https://www.keymusic.com/nl/item/roland-a-49-midi-keyboard-controller-black/]</li>
    <li>Roland EV-5 Expression pedal[https://www.bax-shop.nl/sustain-pedaal-expressiepedaal/roland-ev-5-expressie-pedaal]</li>
    <li>Behringer XM8500 Microphone[https://www.bax-shop.nl/dynamische-microfoon/behringer-xm8500-microfoon]</li>
    <li>XLR to 3.5mm jack[https://www.bax-shop.nl/xlr-jack/stagg-sac3mpsxf-signaalkabel-minijack-xlr](bought at keymusic but was not on their webshop)</li>
</ul>

== Max MSP patch ==
In this section an in-depth explanation will be given for the patch and the program made with it, that is used for the second experiment.<br><br>
Max MSP is a program which lets you combine different objects to create patches that can be used for anything, from audio to video processing.<br>
For experiment 2 it is used to process midi messages and output a new one. A picture of the patch is shown below. The picture is divided into sections. Every section has a function. Notice that every text label in the pictures are in Dutch. Our user is dutch, therefore all the labels are also in Dutch.
<h3>The application behind the scenes</h3>
The first and second section are the places where the inputs and outputs are chosen. Nothing interesting happens there.<br><br>
In the third section, the played note from the input is added to the base note from the guitar string and is outputted to the chosen device from section 2. The add object only outputs a new value if the first input of that object is triggered. Connecting the on-set to that input ensures that only on on-sets the note plays. The user can, i.e. a knob, with Control Change value of 74, to set the standard velocity of the played notes. This means that the notes cannot be played dynamically. Still, the velocity of the played note is routed to this section. This value is then set to either 0 or the value set by the knob.<br>
At the bottom of the section, staff-lines are shown. On these staff lines, the played note is displayed.<br><br>
The fourth section is where the real magic happens. At the top a Control Change signal comes in and routed to the scale object, which inverses the signal. The signal is then send to six split objects where the signal is only send through by one of the six objects based on one of the six ranges. The ranges are also displayed just above the split objects. The split object triggers a specific open string, seen at the bottom. This signal is then send to the add object in previous section.<br>
Two buttons are also present, which put the open strings on the two most used tunings.<br>
Notice how the first and the last ranges are not the same length as the others. This is due to the fact that the pedal is not perfect. During the first couple of degrees of travel there is no signal output. The same goes for the last part of travel when the pedal is pushed down. That is the reason why those two ranges seem very small, while in reality every range is basically the same now.<br><br>
Lastly, in the fifth section a signal is send on closing the window. This triggers a message that stores all values present in all the number objects in this patch. Saving these settings can also be done by clicking the button in this section. Futhermore, on start up all these settings are reloaded using the loadmess object. This object also triggers presentation mode. This mode is explained in the next section. Lastly, here you can also see some text which is used to explain the buttons and numbers in the presentation mode.
<br><br>[[File:MaxBehindTheScenes.png|1000px]]<br>''Click to go to the full resolution''<br>
<h3>Presentation mode</h3>
In this mode, all the wires are removed. Only objects selected by the developer are visible to the user. The sections marked in the picture below correspond to the section in previous picture.<br><br>
In section 1 and 2, the user can easily pick the in- and output devices for using the patch. This is fairly straight forward.<br><br>
In the next section, the note that is played will be displayed. The button above the staff-lines indicates an on-set.<br><br>
The fourth section contains the tuning of the open strings. The tune of a string can be changed by clicking on the staff-lines or using the number box on the bottom. The two buttons at the top of the section will the tuning to the major or minor scale, respectively.<br><br>
In the last section, the user can save the settings of the patch. This mainly includes the tuning of the strings. The user can also view the volume of the played notes.
<br><br>[[File:MaxPresentation.png]]<br><br>