Second project: Educative game for blind children: Difference between revisions

From Control Systems Technology Group
Jump to navigation Jump to search
No edit summary
 
(416 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
= 0LAUK0: Robots Everywhere group 2 =
= 0LAUK0: Robots Everywhere Group 2 =


= Group members =
= Group members =
Line 6: Line 5:
*Axel Deenen (0947031)
*Axel Deenen (0947031)
*Moos Müller (0936214)
*Moos Müller (0936214)
*Dimitar Nikolov ()
*Dimitar Nikolov (1000095)
*Wybe van Vlokhoven (0914565)
*Wybe van Vlokhoven (0914565)


= Project definition =  
= Project definition =
'''After contacting someone who specializes in the technology that had our initial interest, we decided to switch to another subject. Below is what we had done up till then.'''
== Problem statement ==
There are not many resources or convenient ways to practice math and mathematics related skills for blind and visually impaired people. Solving a mathematical problem is largely dependent on the ability to visualize the problem. Without a proper way to visually represent math and observe the whole exercise it becomes very tedious to do problem solving effectively. Take for example exercises on geometry or graphs, the lack of visual aid makes these tasks much harder and sometimes close to impossible as it requires you to remember information you would otherwise be able to look back at. Current methods used to teach blind people are time consuming and expensive, according to our expert Don van Dijk from Visio. These reasons make it less appealing to young blind or visually impaired children to learn math and math related sciences. This leads to an unfamiliarity with math among them while a good mathematical basis is important for everyone.
 
Besides the lack of convenient ways to teach math, there is also a lack of games for blind children that they can play. Most apps and games are made solely for sighted people with lots of visual cues. The blind do however like the few games that they can play very much and would like there to be more available games for them. In this project a solution for both these problems is created.
 
=== Approach ===
The way the final product is made consists of several stages. In the first stage knowledge is gathered. During this stage sources will be found on fields of educational benefits of games, games for blind people but also the way mathematics is taught to the blind. Interviews with an expert on the field of educating blind people will give insight into the possible requirements for the device. During this stage the things that the device should be able to do are defined. After all the requirements are quantified and the desired abilities are listed the second stage can begin. The second stage will be focused on writing the software necessary to fit the requirements. The final product would be an app that will run on a smartphone. During this stage it is checked if all the requirements on what we aim for the end of the course are met and everything works. Below a link to all the sources and the planning can be found.
 
[[Sources and planning]]
 
== State of the art ==
 
[[File:Tactile_printer.png|325px|thumb|right|Figure 1: Tactile printer with an example.]]
[[File:3D-printed graph.JPG|250px|thumb|right|Figure 2: Example of a 3D-printed graph.]]
 
=== Sources ===
The state of the art was first checked by acquiring 25 sources. These sources were about a couple main topics, e.g. teaching mathematics to blind children, educational games, Braille and apps for the blind. This research gave a better understanding of what was important to take into account when making an app, for example voice control on a phone, but mainly the struggles with teaching math to people that are not able to visualize graphs and shapes. The articles also described some of the solutions and devices that were already implemented currently. However, the articles were mainly about the system in the USA and since the project was based in the Netherlands, further research needed to be done. This further research was also useful in another aspect. Some technologies and devices were read about in the sources, but how they were used in practice in schools was still very unclear.
 
=== Visio ===
Therefore we contacted Visio, a Dutch company that works with and for the blind and visually impaired. Through this company we met one of their employees, Don van Dijk, who has experience with teaching math to blind children. Don gave  a lot of useful information. He made clear that the biggest problem in the education of math to the blind is currently that graphs and images are really hard to depict in an understandable way. The company Dedicon can make tangible drawings for them on request, but this takes a long time and could be improved. He also told us that the the Nemeth code is not used in the Netherlands, because another system is developed. Therefore there was no need to use or teach this in the app. Don also gave some useful sites that already had good accessibility for blind users. The best example was [https://www.desmos.com/ desmos.com], a graphical calculator that uses a lot of features for blind users. On this side they use audio trace to visualize a graph, [https://www.desmos.com/accessibility#audio-trace-examples examples can be found here].
 
To get more information about the current systems on schools for the blind and visually impaired, Don offered us a visit to Visio's school in Grave. The tour was given by Don van Dijk himself. It was a very educational excursion where we were shown all the devices available to them and their use in practice. A student taught us how he operated a computer using the 'Brailleregel' for example. Besides this Don explained to us that they used some really old devices that still worked well for them, e.g. a Braille typewriter, and also some newer once.
 
One of the devices that worked really well for them was a tactile printer, a device that helped them explain graphs to their students. Through heating a special piece of paper the black ink on it could create a ridge on the paper. This printer with an example can be seen in figure 1. According to Don this system was most intuitive to the blind and would best help them visualize a graph. Closely followed by a 3D-printed graph, see figure 2. Besides these two ways to some other aids that they used were pin boards, tangible shapes and shapes and graphs created with wire.
<ref>
Willemsen, D.R.S. (2015). Designing Haptic Graphics for Mathematics: Towards Accessible Math Education for Blind Students. TU Delft. Retrieved from https://repository.tudelft.nl/islandora/object/uuid:b613feb9-7460-49c4-bd93-cc917d84f108?collection=education
</ref><br>
 
Don was also interested in the idea of making a game. He explained to us that the games that would last and work best were the games that were made for more users than just the blind. It should have either a function for the blind to be able to play it or not depend on sight at all. The game that was currently popular at their school was a game for iOS called BlackBox. The goal of this game is to complete all levels, and each level is a puzzle the user needs to solve. Little to no hints are given to the user what the puzzle actually is about, meaning that the hardest part is figuring out what needs to be done. This element which made the game unpredictable seemed to be the sort of challenge that the children there could appreciate.
 
== USE analysis ==
The USE analysis will cover how the app will affect the Users, Society as a whole and Enterprises as well. every target group will be addressed separately.
 
'''User'''
 
The primary users for our application are blind or visually impaired children between 6-16 years old. The reason for this is the application topic is geometry and it is at that age that kids start learning about this topic. The fact that humans learn things a lot faster, easier with a more long lasting effect when they are young as opposed when they are adults also plays a big role. Another factor is that the application has the appearance of a mobile game (similar to "Brain quiz" games on the market), therefore being appealing to kids more so than adults.<br>
That is not to say that adults can't play the game.
Adults with visual impairments or blind people can also effectively play the game as it is a unique way of learning about shapes and geometry. They still have the "interest factor" since this is something a lot of blind and visually impaired people don't every get the chance to learn (learning geometry in braille is a very complex, tedious and off-putting to many people). Since this application does not involve any braille it is an alternate way for them to delve in geometry.<br>
And of course "Learning Curve", which is the name of our game, is perfectly suitable for use by sighted people. In this case it still retains the key aspect, where the user has to "sense/feel" the shape or curve on the screen without seeing it while having the menu perfectly visible. Due to the uniqueness of the game it would strike interest in them.<br>
Last but not least teachers that teach to blind and visually impaired children  can use the application by being secondary users. They can try to incorporate it in the teaching of geometry if it shows a good response from the students. This will benefit the teachers indirectly as it will reduce the workload they have and make their job easier and with potentially higher quality in terms of results.
 
'''Society'''
 
The application benefits Society by closing the gap between blind/visually impaired people and people with eyesight. In our modern time a lot of things are developing at rapid speeds but one that always lacks behind, because of its complexity, is the societal imbalance between those two groups of people. Because of their inability to see, blind people do not have access or the means to enjoy many of todays technologies, qualities of life and other aspects, all ranging from basic activities to career and normal social dynamics. While the integration of blind people in modern society has been improving every year, a lot of aspects are hard to overcome. The "Learning Curve" application has the potential to assist in teaching visually impaired people in learning about shapes geometry and therefore reduce the societal gap between sighted and blind people, that occurs on an everyday basis. The imbalance in accessible knowledge between the two groups leads to visually impaired people being unable to be members of certain social sectors (professions that require geometry knowledge like for example). Since our application provides a unique method to learn about geometry we believe it can be a great benefit to society.
 
'''Enterprise'''
 
Since "Learning Curve" is a non-profit free application it provides no direct benefit to the enterprises. It can, however, bring more light on the topic of "Applications for blind and visually impaired", if it gains popularity. This will of course bring competition, as is the case with most innovations. A new fresh market would mean that new competitors will join, wanting to get a stake in it. This is not only beneficial to the companies and developers themselves, but it is also going to lead to further innovations and advances in the sector of "Applications for blind and visually impaired".
 
== Research ==
=== The effectiveness of educative games ===
 
The effectiveness of educative games is largely based on two factors: the educational value of the game and the attitude of students towards the game.
 
'''Educational value'''
 
The educational value of a game is based on how much knowledge is packed into the game and the skills that might be taught by playing the game. “What are the students going to learn from the game?” Is an important question to answer. The game should teach topics which are not easily taught in another form and the time spend on playing the game should be proportional to how much the students learn. It is important to notice that the learning content not only consists of the knowledge taught, several skills are also developed. For instance, gaming could improve problem solving, collaboration, communication and social skills.
 
'''Students’ attitude'''
 
Students have varying types of personalities and therefore all respond differently towards the use of educational games. The seemingly positive effects of games may not be the same for every student. The attitude of the students towards the games is a deciding factor in their behavior. A study of the attitude of students towards an educational game is crucial to its success. A student’s attitude is influenced by four factors: Relevance, Confidence, Media affinity and Self-efficacy<ref name=Marti2017>
Marti-Parreño, José & Galbis-Córdova, Amparo & Miquel, María. (2017). Students’ Attitude towards the Use of Educational Video Games to Develop Competencies. Computers in Human Behavior. 81. 10.1016/j.chb.2017.12.017.
</ref>. These four factors will now be explained in more detail.
 
The relevance of a game should be clear to the student. The relevance is related to the content learned and to the way in which it is taught. If, for instance, a game has a high educational content but the way it is taught is very awkward the perceived relevance is low. On the other hand, if the game is very well designed but it has little to no educational content it would no longer qualify as an educational game. A student should believe the content learned with the game is best learned using the game. The game developed in this context might have relevance to the student because it will help them visualize a graph or shape in different ways. By using various sensors in their phones this way of visualization may only be achieved by the described game.
 
The students’ confidence in succeeding will influence the students’ persistence. They should not be worried about their inability to properly use a game to learn. Frequent feedback in the form of rewards during the game will increase their confidence and various game levels (easy, medium, hard) will allow the student to learn at their own pace. The graph game will increase the user’s confidence by giving of both visual as audible rewards after completing a level. It will also remember the amount of levels that were completed to stimulate a sense of achievement.
 
Media affinity is the importance that a medium has in the lives of the user. Research suggested that, for example, the affinity towards the use of a mobile phone had a positive influence on mobile shopping<ref name=Marti2017 />. The iPhone and other Apple products are of major importance to visually impaired and blind people, as was told to us by Don van Dijk. These products help them to function as normally as possible by the use of several features such as voice assist, resulting in a high media affinity.
 
Self-efficacy is the belief of an individual in their own abilities to achieve a desired outcome. The student has to believe he/she has the ability to finish the game and to gain the desired knowledge to play the game. Self-efficacy might, in this case, be influenced by eliminating the possibility of failure and letting the student get acquainted to the game by the use of a simple tutorial.
 
=== Considerations on entertainment ===
 
As explained  in the section “on the effectiveness of educative games”, the amount of new things that a student learns from an educative game should be proportional to the time spent playing the game. Clearly then, it is desirable that the game is played extensively by the student. <br>
This desire can be fulfilled by exploiting one property of games: their goal is not only to educate, but also to entertain. If this last characteristic is implemented properly, the students will voluntarily occupy themselves with the game. <br>
 
''' Mechanisms of entertainment'''<br>
 
The question then becomes: how can an educational game be made entertaining, without comprising its educational core? Perrotta et al<ref name=Perrotta2013>
Perrotta, C., Featherstone, G., Aston, H. and Houghton, E. (2013). Game-based Learning: Latest Evidence and Future Directions (NFER Research Programme: Innovation in Education).</ref> have done extensive research into videogame based learning, and have listed a number of principles which should be ingrained in the design of educational games, as well as the mechanisms by which they can be implemented in a game. Below are listed the mechanisms which can provide entertaining value.<br>
 
'''Challenging goals'''<br>
 
Providing a clear goal will show the users the result of their effort. It should be clear towards what the users are working. The goals should be quite challenging, or else the users won’t take the game seriously or they will get bored. <br>
 
Our game could implement this by having a number of levels, each pertaining to a different function and corresponding graph. If the game is too easy for a user, he or she can advance to levels using more difficult functions. <br>
 
'''A fictional setting to provide a compelling background'''<br>
 
Perrotta notes that this feature should not be used as a means of escapism for the users, but should be used to provide them with an environment in which they feel free to experiment with different methods of approaching a problem, without risking actual failure as in real life. <br>
 
'''Difficulty levels'''<br>
 
Using different difficulty levels through which the users can advance progressively will stimulate and challenge the users to improve their performance. Again, the criteria for advancing or “leveling-up” should be made explicit. Advancing to a next level can be combined with a sort of reward, thereby acknowledging the user’s mastery of the material. This also increases the feeling for the users that they are in control, and are directly responsible for their performance. <br>
 
'''Uncertainty'''<br>
 
Adding a non-linear element, such as the ability for the users to choose different tasks themselves instead of always being presented with a task, also increases the user’s feeling of control. Doing so will encourage the user’s to think independently and with that, increase their capability of learning new things. <br>
 
Users should be able to select themselves different levels in our game. They won’t have preliminary knowledge about the actual contents of a level, but it would be possible for them to know something about the relative difficulty of a given level.
 
'''A social element which allows the users to share experiences'''<br>
 
Another advantage of games is they allow for a strong social experience. If users are encouraged to share their findings and knowledge about the game so that others can advance, users may form bonds. This will increase the enjoyment users experience from the game, besides allowing them to develop additional social skills. <br>
 
An online ranking system could be added, so that users can compare their performance with their friends.
 
=== Possible ways a phone can be used to sense a graph ===
If our game is to be in the form of an app, we would like to use the full input potential of current smartphones. Below is a list of different ways users could perform different tasks, with respect to a game involving graphs.
 
Line to formula
*The user traces the graph by moving his/her finger along the line. If the user’s finger diverts to much from the line a vibration will be produced. The further away once finger from the line the harder the phone will vibrate or the shorter the time between vibrations.
*The user follows the x-axis and the higher the y-value the higher the pitch of the sound that is being produced (DESMOS). Or the user traces the graph and the further away the finger is the higher the pitch.
 
The levels below have to be performed in a controlled environment. The user gets the formula of a graph through audio and visual signals. It uses a combination of all the sensors in a phone.
 
Formula to line
*The user has to move the phone according to the shape of the graph.
* The GPS is used to calculate the speed at which the user is moving. This allows the user to move in a straight line while accelerating and decelerating according to the shape of a graph.  A good way to learn derivatives.
* The user has to move in the shape of the graph. The movement is traced by the GPS.
* The user draws a graph on a piece of paper and takes a photograph of it. Recognition software will check if the right graph is drawn.
* The user sings through the microphone at the right pitch to represent the height of the graph.
 
=== Description of the educative game ===
 
The game consists of a collection of levels, which do not necessarily have to be completed in order. The ultimate goal of the game is to complete all the levels. By completing levels the users gains points which the user can trade for hints, new levels or new sounds. There are two game modes that will be imlemented in the game. The goal of the first game mode is to figure out what function it is that a level contains by tracing the graph of that function. The goal of the other mode is to draw a graph corresponding to a specified function. However, each level represents the graph or requires its input in a completely different way, but it is certain that in none of the levels the graph will be visual on the screen. The users will need to use various senses and methods to try to visualize the graph. They will also need mathematical intuition and knowledge to connect the shape they have in mind to a function, or the other way around.<br>
 
The core learning part of the game is to gain “a feeling” for the behavior of functions and shapes. As stated, users will receive no information whatsoever on how to solve a level. This is much the same as with the app Blackbox, which was according to an expert from Visio a big success among the blind and visually impaired children (as well as seeing children). Since users do not even know how to describe the graph, they will have to use all possible ways of inputting information they can think of. It is this that will challenge the creativity of the user, and also be the most enjoyable part of the game. Herein lies also the biggest design challenge, since all levels need to be more or less unique to keep the game challenging and surprising.<br>
The game will also contain an online ranking as to encourage users to play it with friends and help each other solve problems. This can be done as follows. When users complete levels, they can gain points. The amount of points they get will be proportional to their performance, measured by a number of parameters such as speed and accuracy. For example, users could get upon completion of a level either one, two or three points. If they complete it without mistakes, i.e. correctly following the shape of graph at all times, and within a certain time limit, they get full points. A point can be subtracted if between one and three mistakes are made, and only one point is awarded when more than three mistakes are made.<br>
 
Users can see a ranking of their performance based on the total number of points they have, compared with other players. In addition, if a user completes a level with three stars, that user gains a point. This point can be, as described above, used to obtain ornaments such as new sounds, or it can be used to enable more functional features such as new levels or hints on how to complete an existing level.<br>
 
In addition, users can assist their friends not only verbally, but also by donating them points, so their friends can unlock hints for difficult levels.
 
= Solution =
The Solution section is split in two parts the "Objective 1" and "Objective 2". Objective 1 elaborates how we envision the solution being done given we had enough time, knowledge and resources to work with, and "Objective 2" represents what we were able to achieve during the span of the course. Everything is written in such a way that this idea can be easily expanded and the full functionality of the app can be realized.
 
== Objective 1: Create a concept of an app that teaches the behavior of graphs to blind children ==
=== Objectives ===
The End product idea that we envision is an app called "Learning curve", which engages blind children in a light-hearted manner with the visualization of graphs.
<br>
The app should provide children with a brief explanation of a mathematical function. This explanation can be read out loud by the app. This is to give the child the necessary conceptual knowledge. The core of the app will be a game in which children are challenged to apply their creativity and knowledge of functions to try to figure out which graph it is they need to identify. In short, we aim to make an app which turns what is ordinarily a tedious and slow process into something enjoyable for blind children.
 
=== Requirements ===
''Note: These are the requirements that we envision for a proper end product version of our app, i.e. for objective 1.''
 
*A tutorial needs to be present that introduces and teaches various functions and their corresponding graphs.
*A game which challenges the player to visualize functions and understand their shape.
*The option to play with other players online.
*Incentives to keep the player engaged with the game on a long term (gain points for playing and more if one does well, online ranking so one can compare himself/herself with other players).
*Game needs various functions such as audio and vibration.
*A practice mode that is single player and can be played offline.
*Multiple difficulties where more complex functions are used on a higher difficulty.
*Input from the user is obtained through various ways such as a touch display, microphone and accelerometer.
 
[[File:Structure level select.png|200px|thumb|right|Figure 3: The structure of level selection.]]
 
=== Features ===
 
The end product is an iOS app that can be downloaded from the app store. As discussed earlier the final product should be made for Apple since most blind children use Apple products. There are a few features necessary to make the app both educational and enjoyable.
 
[[File:Structure different formulas 2.png|200px|thumb|right|Figure 4: The structure of different functions in level selection.]]
 
Suggestions for the different levels to be implemented are described in the "Possible ways a phone can be used to sense a graph".
 
Since navigating through the app is important (think of choosing levels, using hints or changing settings) the menu's have to be specially designed to be accessible to blind, visually impaired and sighted users. Navigation through the app will make use of the 'voice over' function. This voice over function has the ability to read the text written on a button on the screen of the phone when the button is clicked. Only when the button is clicked again will it trigger the button. If, however, the user click anywhere else on the screen the button will reset and the user has to double click it for it to trigger. Everything clickable in the game will have sound, and buttons with similar functions, such as a button for going back to level select and a button for going back to the main menu will have similar sounds, but with different lengths and pitches. This will allow blind users to navigate the game easily, but the sounds also serve an important role in creating an enjoyable and immersive game enviroment<ref>Ekman, I., Ermi, L., Lahti, J., Nummela, J., Lankoski, P., & Mäyrä, F. (2005, June). Designing sound for a pervasive mobile game. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology (pp. 110-116). ACM.</ref>. Furthermore, the buttons will all be fairly big to ensure that clicking the right thing is easy and visually impaired users can see them better. The buttons will also contain written text. The main menu will contain five buttons: level selection, explanation, online ranking, shop and options.
 
The level selection button will take the user to the next screen in the app in which all the levels are displayed in the form of icons with numbers in them. When a level is clicked on it will not only state the name/number of the level but also if it has been completed already by the user, to help the user keep track of the progress made (of course the user is free to complete a level multiple times). When a level is selected the user will be send to a next screen which is build up to match the desired functions of the level. An example of a level could be a level in which the users has to move its phone through the air, trying to follow the shape of the unknown graph. The more the shape traced with the phone resembles the graph the longer and higher a pitch will sound. When the path traced through the air with the phone resembles the graph enough (correct amount of maxima and minima, correct qualitative increase or decrease in slope) the level is completed. It could take quite a while before the user figures out how to "see" the graph. Before the user knows the movement of the phone is the input, the user might experience moments in which the game notifies the users that the answer is wrong, while the user does not know why he receives that message. In this exploration aspect lies the fun of the game. Trying to think of functions to solve the level, while not having all possible obtainable information could also learn the user a lot about the graphs of functions. Only if the player has given the right answer in a level, the next level will be unlocked and the user is taken back to the level selection menu. Points will be reward after completing a level. Giving a wrong or right answer will trigger an appropriate sound.
 
Figure 3 shows the structure the level selection menu will have. A level starts off by tracing the graph on the phone, followed by answering a multiple choice question to check if the user understood what function he just traced. After answering the multiple choice question correctly the two 'drawing the graph' levels unlock. These drawing levels are two options from the list in the 'Possible ways a phone can be used to sense a graph' chapter. For example this can be drawing the graph on paper or having to walk the graph. To the user it is unknown which options the levels use and that is what he/she has to figure out. The options are different for every new graph he/she unlocks.
 
Besides the two drawing levels, after completing the multiple choice question, the next function also unlocks. This is shown in figure 4. The next graph unlocks after answering the multiple choice question correctly, because then the user can choose to continue to next graph if he/she does not want to or cannot complete the drawing levels at the time he unlocks them. In figure 4 it is shown clearly that the multiple choice levels and the drawing levels are on different paths. The further the user goes from the first graph level the harder the functions are. This increases the difficulty over time for the user when he is more familiar with the workings of the app and has a better visualization of the easier functions.
 
The explanation button gives the user the needed knowledge to complete levels. It will contain some general hints and possible levels they can encounter. This way they know what to expect from the levels and the game as a whole.
 
An online ranking button will be added to give the users the drive to compete. The button will take the user to a list with other users. The list can contain global users, national users or a closed group of users (think of a list of classmates). After opening the online ranking the voice over will tell the user their current rank. Hereafter, the entire list will be read out. On the screen below there will be a skip button which, after clicked, skips a desired amount of people (this can be set in the settings).
 
The points scored can be used in the shop. This shop will contain the possibility to purchase sounds, different colors or even levels with the point system. Spending points will not effect the ranking of the player.
 
In the options menu the settings of the game can be adapted. The voice over function can be toggled on and off, a high contrast mode can be selected and the online ranking settings can be adapted. Also the language can be changed here. Some basic colors are already available here to adjust to different types of visually impairment. This way they can adjust the colors to the once that makes it the easiest for them to see.
<br><br><br><br><br><br>
 
== Objective 2: The achieved game ==
 
=== Requirements ===
 
Due to lack of time, the actual app will not fulfill all the requirements listed under objective 1. Instead, we aim to create an app which incorporates some key elements of the above mentioned properties. This will allow for a proof of concept, i.e. see to what extent an app focussing on learning the behavior of functions will be helpful for and appreciated by blind children. The requirements that are needed for such research are listed below.
 
*The app needs to be navigable without any visual information.
*The app must contain a visual interface, so that it will be accessible to sighted people as well.
*The app needs to contain one fully functional level, including the depiction of a specific function, and the option to answer which function it is. This also encloses the possibility of giving a wrong or right answer, and the app needs to indicate whether a wrong or correct answer is given.
*The app must have a general user interface, which includes a main menu from where the user can choose to play a level.
 
=== Deliverables ===
 
Since time is limited the goal of our project is to create an Android game app as described above containing at least one working level. Our initial idea for a solution to the problem was a game in the form of an iOS app. This platform, instead of Android for example, is chosen because it was found that the majority of blind people own an iPhone due to it having very user friendly capabilities for blind users. However, after some delibration, it was decided that the prototype of our app will be made for Android. The reason for this is purely practical, in the sense that none of the group members possesses or has any means of acquiring a device which runs iOS. This doesn't detract from the value of our project, since we aim only to build a single level to test the feasability of an educative math game which is meant specifically for gaining intuition for the shape and general behavior of graphs. Also we believe that the platform is of little importance as we can achieve the desired effects on Android devices as well. All this will be discussed in further detail below. The game is also designed such that it can be played both by seeing, visually impaired and blind people. This is done to create broad support for the app, as advised by an expert from Visio. The app to be delivered does not contain all the features described in objective 1. It will not contain the point system, the shop, the online ranking and an options menu. Nor will it contain the settings or a wide variety of different levels. The code is, however, written in such a way that it can easily be extended to fulfill all the features.
 
The game will only have one gamemode, namely to recognize the graph and answer the multiple choice question. Therefore the level select screen contains only a grid
of levels instead of a distinction between the different game modes. The game will also contain only one level type, as in that the vibrating function of the phone will be used, in all three levels, the indicate the shape of the graph. But this does not result in a change in the level select screen compared to the theoretical concept of the app.
 
=== The developed app ===
 
[[File:app_logo.png|400px|thumb|center|The logo of our app "Learning Curve".]]
 
'''Main menu'''
 
The first screen the user will encounter when opening the app is the main menu as seen in the screenshot in figure 5. The main menu contains five large buttons labeled: “level selectie”, “uitleg”, “online ranking”, “winkel” & “opties”. When one of these buttons is tapped the text on the button will be read out loud by the phone. Only when a button is clicked twice without clicking on another button in between will it perform an action and play a clicking sound similar to that of a chest being opened. The only button in the main menu that leads to another screen is “level select” and because of this it is the only button that will play the clinking sound when clicked twice in a row. Since the app is meant primarily for blind dutch schoolchildren, the interface is in Dutch.
 
'''Level selection'''
 
The level select screen contains twelve square buttons in a grid all labeled with a level number as seen in figure 6. When first opening the app only level 1 will be the same bright blue as the rest of the app to indicate to the sighted users that this is the only unlocked level. When level 1-3 are clicked once the number of the level (as well as the word level) will be read out loud. When the user clicks again on level 2-3 the app will tell the player that this level has not been unlocked yet. When the user clicks again on level 1 it will enter level 1 and play the clicking sound. If the user clicks on level 4-12 the app will tell the user that “this level is not available in the demo”. Only level 1-3 are functional and each time a user completes a level the text of the next one will be turned to the bright blue and it will be unlocked as can be seen in figure 7.
 
'''The level'''


A huge amount of information is communicated to people through text. Many of that is not obtainable for fully blind or near-blind people. It is possible to make a large portion of this information accessible by designing a portable device to assist the blind. This device would be able to scan text and present it in braille code. The goal is to make it easy for the blind to read books, hand written texts and text contained within images. A hypothetical concept for this device will be designed for fully blind people.
Again the goal is to find the function belonging certain graph, where the graph can be represented through various ways which are unknown to the player. In this level the graph will be communicated to the player through the touchscreen and vibration function of the phone. The player can move its finger across the screen of the phone and if the player is within a certain distance to the graph then the phone will vibrate. Otherwise it will not. Now that the player can trace the curve he can visualize the function. When ready the player can click on a button in the bottom of the screen. When clicked the player can choose from 4 multiple-choice answers which will be located in the four vertical quadrants of the screen of the phone. The answer will be read out loud when clicked on, and will be chosen when clicked on again (as is with all buttons). When the player chooses the wrong answer the level needs to be solved again, but with a different graph. It is to be noted that in our case we only have one graph hence selecting a wrong answer will not result in a new graph rather the same one. The level screen contains only one button named “antwoorden” as seen in figure 8. This button is to be clicked (again twice) by the player when he/she thinks to know the answer. When the user touches the (black) part above the button of the screen the phone will vibrate unless the finger of the user is within a certain (shortest) distance to the graph. The user will need to figure this mechanic out for themselves, as it is part of the fun exploration in the game the blind children enjoyed. The app contains three working levels with the functions y = 0, y = x & y = x^2 (in that order).


==Approach==
'''Answering'''
At first literature will be studied to learn about the current technological aids for the blind and for the current state of OCR (optical character recognition) and refresh-able braille displays. Then contact with a institute, involved in research for helping the blind, will be sought to gain feedback on the device. This feedback will then be carefully incorporated into the device to ensure its practical use.


==Results from literature research==
The answering screen contains four button labelled each with a different function as seen in figure 9. Again the voice-over double click function is implemented. When the user clicks the wrong answer a loud buzzer will sound indicating that it was wrong and the user will be redirected back to the level the user was trying to answer so he/she can try to visualize the graph better. When the correct answer is selected a cheerful harp will sound and the user will be redirected back to the main menu.
After conducting a vast literature research, a brief summary of all articles and patents deemed relevant has been made. These summaries are listed below.


'''A BRAILLE O.C.R. FOR BLIND PEOPLE'''<ref>Hermida, X. F., Rodríguez, A. C., & Rodríguez, F. M. (2008). A BRAILLE O.C.R. FOR
{|style="margin: 0 auto;"
BLIND PEOPLE</ref><br>
| [[File:app_main_menu.png|230px|thumb|upright|alt=fig5|Figure 5: The main menu screen.]]
In this paper the principles for an optical character recognizer, OCR, for written braille code are developed. Several image processing techniques are incorporated such as adaptive thresholding and skew angle detection. The paper was sponsored by the National Organization of Spanish Blind People.
| [[File:app_level_selection.png|230px|thumb|upright|alt=fig6|Figure 6: The level selection screen when first opening the app.]]
| [[File:app_level_selection_2.png|230px|thumb|upright|alt=fig7|Figure 7: The level selection screen when level 1 has been completed.]]
| [[File:app_the_level.png|230px|thumb|upright|alt=fig8|Figure 8: The level screen.]]
| [[File:app_answering.png|230px|thumb|upright|alt=fig9|Figure 9: The answering screen.]]
|}


'''OCR-OPTICAL CHARACTER RECOGNITION'''<ref>Verma, A., Arora, S., & Verma, P. (2016). Ocr-Optical Character Recognition. 7th
=== Explanation of the code ===
International Conference on Recent Innovations in Science, Engineering and
Management, 230–240.</ref><br>
In this paper a simple, efficient and less costly OCR for regular text is presented. The OCR uses a database to recognize the characters to lower the computational cost of the program. The paper discusses various stages of the program including preprocessing and post processing.


'''Portable Braille Computer Device'''<ref>Kahn, S. (2003). PORTABLE BRAILLE COMPUTER DEVICE.
The code works by defining two arrays corresponding to the x- and y-values of the graph. The number of elements in the x- and y- array are both equal to the width of the screen in pixels. The standard coordinate system on the phone (which has the origin in the left corner, the positive y-axis pointing downward and the positive x-axis pointing to the right) is converted to a coordinate system with the origin in the center of the screen. All the x-array elements are filled in in such a way that the first element corresponds to minus half the width of the screen and the last element is plus half the width of the screen. These x values are then used to the define the y values in the y-array according to a desired function.  
doi: 10.1126/science.Liquids</ref><br>
This patent describes a portable device that can scan text using a scanner, interpret the scanned text using an OCR program and present it using a braille display. The patent also claims to provide a complete information management system within the device. Such as a personal agenda and notes.


'''Wearable assistive devices for the blind'''<ref>Velazquez, R. (2010). Wearable assistive devices for the blind. Lecture Notes in Electrical
The position of the user’s finger is then measured. A circle can now be drawn around this position with its radius corresponding to a desired margin. If the graph is not within this circle and the user moves his/her finger, the telephone vibrates. The vibration last for 5 seconds unless something else happens that will automatically terminate the vibration (i.e. if the user lifts his/her finger). The vibration will not restart every cycle because the program remembers whether it is on or not. This is achieved by the use of an array consisting of two elements, one for the current loop and one for the loop that was done before. If there is no change between the current loop and the loop before it, there will be no change. If they differ the vibration will be started or cancelled accordingly.
Engineering, 75 LNEE, 331–349. doi: 10.1007/978-3-642-15687-8-17</ref><br>
This paper reviews important steps made in the field of wearable assistive devices for the blind. From these important steps universal design concepts for these wearable devices, and also other systems for the blind, are extracted. The future potential of various prototypes are also discussed.


'''A Blind Person’s Interactions with Technology'''<ref>Tenenberg, J. (n.d.). A Blind Person’s Interactions with Technology.
=== App download ===
COMMUNICATIONS OF THE ACM, 52 (8). doi: 10.1145/1536616.1536636</ref><br>
This paper considers a blind person's interaction with various technologies. The technologies are discussed both practically and socially, since both are important when using it in one’s daily life. Findings about the interaction with technology of a person blind from birth are presented.


'''An interactive and multi-functional refreshable Braille devicefor the visually impaired'''<ref>Fatih Basciftci, A. E. (2016). An interactive and multi-functional refreshable Braille device for the visually impaired. Displays, 41 , 9.</ref><br>
Please feel free to test our android app on your own phone! Below is a .zip file containing the .apk file that you can install on your android phone. Do not forget to turn on “Allow installation of apps from unknown sources” in the security tab within the settings of your phone. And please note that the app might not function 100% correctly on every phone, since it only was tested on a couple of different phones.
The authors of this paper have developed a book reader with 96 braille cells, and also includes to option to convert text to audio. The articles describes in depth the design of the device, including all the buttons and their functions. It uses a scanner which sends information to the device via USB or Bluetooth, which then converts the information to braille. Tests done by the authors with visually impaired people that can or cannot read braille indicate a high usability of this device. Overall, this device is very similar to our initial conception.


'''Methods for presenting braille characters on a mobile device with a touchscreen and tactile feedback'''<ref>Rantala, J., Raisamo, R., Lylykangas, J., Surakka, V., Raisamo, J., Salminen, K., Hippula, A. (2009). Methods for presenting braille characters on a mobile device with a touchscreen and tactile feedback. IEEE Transactions on Haptics, 2 (1), 28–39. doi: 10.1109/TOH.2009.3</ref><br>
[[File:Learning Curve.zip]]
In order to circumvent the high cost of a mechanical braille reader (usually in the range of 5000 to 15.000 USD, according to the authors), different methods for implementing braille reading in a mobile device were researched. The authors identified three different ways of presenting braille characters, and showed the feasibility of all methods through experiments in which participants were asked to identify a single character. Both the accuracy and reading time  varied, and the authors stated that further research would be required to find out how to present words or sentences.


'''Determining the optimum font size for braille on capsule paper'''<ref>Watanabe, T. (2014). Determining the optimum font size for braille on capsule paper.
Also feel free to improve on our app or have a look at the code in the zipped Android Studio project folder below.
IEICE Transactions on Information and Systems, E97-D(8), 2191–2194. doi:10.1587/transinf.E97.D.2191</ref><br>
This paper presented research on what is the optimal size of a braille front. The results showed that there was an optimum size, which gave the highest reading speed and the lowest error. The optimal size showed little variation for different age groups. The most important conclusion to be drawn is that enlarging the font size doesn’t necessarily improve either reading accuracy or reading time.


'''Camera Reading for Blind People'''<ref>Neto, R., & Fonseca, N. (2014). Camera Reading for Blind People. Procedia Technology,
[[File:Learning_Curve_Android_Studio_Project.zip]]
16 , 1200–1209. Retrieved from http://linkinghub.elsevier.com/retrieve/pii/S2212017314003624 doi: 10.1016/j.protcy.2014.10.135</ref><br>
This article describes the foundation for a technique that should enable blind people to photograph a piece of text using a mobile device, and subsequently have that device read out the text. The paper present an overview and state of art (from 2014) of Optical Character Recognition (OCR) and Text-To-Speech (TTS) software.  


'''A Review on Optical Character Recognition Techniques'''<ref>Modi, H., Scholar, P. G., & Parikh, M. C. (2017). A Review on Optical Character
=== User test case ===
Recognition Techniques. International Journal of Computer Applications, 160 (6), 975–8887. Retrieved from http://www.ijcaonline.org/archives/volume160/number6/modi-2017-ijca-913061.pdf</ref><br>
This paper describes more in depth the process of OCR. The steps through which an OCR protocol goes are subdivided into digitization, pre-processing, segmentation and feature extraction.


'''Computerized microfluidic cell culture using elastomeric channels and Braille displays'''<ref>Gu, W., Zhu, X., Futai, N., Cho, B. S., & Takayama, S. (2004). Computerized microfluidic cell culture using elastomeric channels and Braille displays. Proceedings of the National Academy of Sciences, 101 (45), 15861–15866. Retrieved from http://www.pnas.org/cgi/doi/10.1073/pnas.0404353101 doi: 10.1073/pnas.0404353101</ref><br>
Because we did not know if the user test could take place before the deadline of the project, two test cases were made. One preferable situation were the test was done with a blind student and a teacher and a back-up test case done with blindfolded volunteers.
This article describes a new computer-controlled microfluidics system used on a refreshable Braille display. It is a grid of 320 vertical pins to depict the braille. Due to a new method used to power the integrated pumps and valves, controlling the pins, in the silicone rubber the system is a lot more efficient. More pumps and valves can be put in a smaller space improving the display immensely.


'''Optical Character Recognition for Handwritten Cursive English characters'''<ref>Aparna, A., & Muthumani, P. I. (2014). Optical Character Recognition for Handwritten
'''Ideal situation'''
Cursive English characters. International Journal of Computer Science and Information Technologies, 5 (1), 847–848.</ref><br>
This paper talks about OCR for handwritten text. To improve OCR for handwritten text a new method is developed with noise cancelling. A median filter is applied to filter out unwanted scribbles to improve the results. To get the results the following steps are used: image acquisition, image preprocessing, segmentation, feature extraction, recognition.


'''Real-time scene text localization and recognition'''<ref>Neumann, L., & Matas, J. (2012). Real-time scene text localization and recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 3538–3545. doi: 10.1109/CVPR.2012.6248097</ref><br>
The game is tested with blind children at the Visio Onderwijs Grave. A phone is given to the children with the app on it. The app is started already, since it is made on an android device and not on iOS. Because of this the app list cannot be read aloud. After the app is started the children should be able to operate the app without any assistance and be able to answer the questions correctly. Afterwards a few questions are asked. They are as follows:<br>
An article about text recognition. It discusses a method for real-time text recognition on images for example. It creates an Extremal Region (ER), a box, around the text region to include the entire text without missing anything, while not including areas where no text is present. The system is tested by two methods. The ICDAR 2011 and the Street View Text dataset.
Is the app intuitive?<br>
What is the shape you just followed?<br>
Do you think this can help you learn math functions?<br>
Do you have tips on how to improve the app?<br>


'''A system for converting print into Braille'''<ref>Blenkhorn, P. (1997). A system for converting print into Braille. IEEE Transactions on
The following questions are asked to the teacher.<br>
Rehabilitation Engineering, 5 (2), 121–129. doi: 10.1109/86.593266</ref><br>
Do you think this apps helps the students?<br>
The paper goes over different methods of braille that are in use. It adds certain language rules to make converting text to braille easier. It does however also state that the rules should be developed further.
Could you see this being used to assist teachers?<br>
Do you think the app is useful?<br>
What could be done to improve the app?<br>
Do you have tips on how to improve the app?<br>


'''REFRESHABLE BRAILLE DISPLAY SYSTEM'''<ref>Schmidt, R. (2002). REFRESHABLE BRAILLE DISPLAY SYSTEM. Patent no: '''US 6,354,839 B1'''</ref><br>
'''Less ideal situation'''
This is a patent of a refreshable braille display system. It is a display that can extend and retract dots to form words or short sentences. It describes a microelectromechanical device.


'''Voice Assisted Text Reading System for Visually Impaired Persons Using TTS Method'''<ref>Sanjana, B., & RejinaParvin, J. (2016). Voice Assisted Text Reading System for Visually
In case the testing with the blind children is impossible another user test case should be held. In this case testing should be done on blindfolded volunteers with no understanding of the workings of the app. Again a phone should be given with the app launched, but besides stating that the goals aims to teach math, no instructions should be given. The blindfolded test subjects should be able to operate the app on their own. The only assistance allowed is translations for non-dutch speaking subjects. After the use of the app some questions should be asked.<br>
Impaired Persons Using TTS Method. , 6 (3), 1–5. doi: 10.9790/4200-0603031523 </ref><br>
Is the app intuitive?<br>
In this article the TTS method is elaborated. A finger mounted camera is used to capture the text image from the printed text and the captured image is analyzed using OCR. A predefined dataset is loaded in order to match the observed text with the captured image. Once it is matched the text is synthesized for producing speech output. MATLAB is used for performance analysis.
What was the shape you just followed?<br>
Do you think this can help visualize a function for blind people?<br>
Was the lack of sight during the test an issue?<br>
Do you think this can help teach math to blind people?<br>
Do you have tips on how to improve the app?<br>


'''Eight-dot Braille'''<ref>Dixon, B. J. (2007). Eight-dot Braille. (September).</ref><br>
'''Results'''
Elaboration on the use of 8 dot braille. It can be used to give more information per cell and it opens a way to produce a wider variety of symbols. It might be used to determine the users location.


'''FingerReader: A Wearable Device to Support Text-Reading on the Go'''<ref>Roy Shilkrot, J. H., Connie K. Liu, P. M., & Nanayakkara, S. (2014). FingerReader: A
The test of the ideal situation was done with a 16 year old blind student named Tom and Don van Dijk. They both liked the idea of the app a lot and were very enthusiastic. Don found it however regrettable that there would come no final product from this project, because the idea and demo were interesting to him. The test case did have some issues with it and could not be done completely independent. The navigation of the menus was harder than imagined, therefore Tom needed help selecting the levels. The level itself was played completely independent. The difficulty of the navigation came with locating the buttons. There was no good solution for this implemented into the app. The text-to-speech helped him, but finding the buttons was hard.<br>
Wearable Device to Support Text-Reading on the Go. In Proceedings of CHI ’14 Extended Abstracts on Human Factors in Computing Systems(Vi). doi: 10.1145/2559206.2581220</ref><br>
These were the answers Tom gave us after playing the levels:<br>
The article describes a finger worn device with a camera which allows the user to skim through a page. This camera records the text and sends the images to a computer chip to convert the text into speech. It uses fibration to correct the users movement.
(Note: these answers are reproductions of his answers and not direct quotes.)<br>
Is the app intuitive?<br>
Yes. The navigation of the menu less so, because it was different than what he was used to. The level itself felt however very intuitive.


'''Character recognition — A review'''<ref>Govindan, V. K., & Shivaprasad, A. P. (1990). Character recognition — A review. Pattern Recognition, 23 (7), 671–683. Retrieved from http://www.sciencedirect.com/science/article/pii/003132039090091X doi: 10.1016/0031-3203(90)90091-X</ref><br>
What is the shape you just followed?<br>
This article elaborates on some of the OCR techniques. Due to the age it might not be the most relevant information.
In order a line, a slope and a parabola.


'''Design and Implementation of OCR to identify English Characters and Numbers'''<ref>Adhvaryu, R., Parikh, R., & Vora, K. (2018). Design and Implementation of OCR to
Do you think this can help you or younger students learn math functions?<br>
identify English Characters and Numbers. , 4 (2), 57–62.</ref><br>
Yes, he understood what shapes he followed and could visualize them.
This article describes the matrix matching method of OCR. According to the article there are two distinct methods for OCR: Matrix Matching in which the system compares the scanned characters with the library character matrices. This system works best when the characters to be scanned and the library characters have very little or no variation in style. Feature Extraction: Feature Extraction generally deals with the features of characters like shape, closed areas, diagonal lines, line interaction and curves. It is more effective and flexible methods as it has a wide scope to identify the same character with different shapes and dimensions. It also gives a short description of how a Matrix Matching OCR is build up.


'''Display of virtual braille dots by lateral skin deformation: feasibility study'''<ref>Lévesque, V., Pasquero, J., Hayward, V., & Legault, M. (2005a). Display of virtual braille
Do you have tips on how to improve the app?<br>
dots by lateral skin deformation: feasibility study. ACM Transactions on Applied Perception, 2 (2), 132–149. Retrieved from http://portal.acm.org/citation.cfm?doid=1060581.1060587 doi: 10.1145/1060581.1060587</ref><br>
No.
This article describes an attempt to make a display for blind people in braille. It explains how lateral skin deformation works and how it can be applied to a keyboard.It gives a good guideline on how certain things work and how they can be used to benefit reading braille.


'''Display of Virtual Braille Dots by Lateral Skin Deformation: A Pilot Study.'''<ref>Pasquero, Jerome. Levesque, V. (2004). Display of Virtual Braille Dots by Lateral Skin
The following questions are asked to Don.<br>
Deformation: A Pilot Study. Proceedings of Eurohaptics, 96–103.</ref><br>
(Note: these answers are reproductions of his answers and not direct quotes.)<br>
This article addresses the differences, advantages and disadvantages between VR information delivery and normal plain paper braille. While Digital or VR based reading has its downsides its positive impacts on the process of blind people reading is uncontested.  
Do you think this apps helps the students?<br>
To give a definitive answer the app should be tested further and for a longer time in a class. After this it can be clearly said if the app helped or not. First impression of the demo is however positive and he could see it being helpful. Every way to visualize graphs to the blind is helpful.


'''V-braille: haptic braille perception using atouch-screen and vibration on mobile phones'''<ref>Jayant, C., Acuario, C., Johnson, W., Hollier, J., & Ladner, R. (2010). V-braille: haptic braille perception using atouch-screen and vibration on mobile phones. Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility - ASSETS ’10 (January), 295. Retrieved from http://portal.acm.org/citation.cfm?doid=1878803.1878878 doi: 10.1145/1878803.1878878</ref><br>
Could you see this being used to assist teachers?<br>
This article is about a electronic based reader for blind people. Instead a device that is mechanical they use a display based on touch as well as vibration from the screen of a mobile phone for example. It's an interesting idea.
Yes, after further testing and development. See previous answer.


'''Slide Rule: Making mobile touch screens accessible to blind people using multi-touch interaction techniques'''<ref>Kane, S. K., Bigham, J. P., & Wobbrock, J. O. (2008). Slide Rule: Making mobile touch screens accessible to blind people using multi-touch interaction techniques. Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility (Assets ’08), 73–80. Retrieved from http://dl.acm.org/citation.cfm?id=1414487%5Cnhttp://dl.acm.org/citation.cfm?id=1414471.1414487 doi: 10.1145/1414471.1414487</ref><br>
Do you think the app is useful?<br>
This article is a bit different because it addresses an innovative way to interact via touch screen with a device. It explains about a new touch screen technologies that allows for simple movements of the fingers (hand gestures done on the screen) to be used to navigate menus. This i feel could be very useful considering the implementation of buttons is not something convenient when making devices for blind people. This could serve as the way we control the device.
It could be useful.


'''MobiReader : A Wearable , Assistive Smartphone Peripheral for Reading Text'''<ref>Polanco II, M. (2015). MobiReader : A Wearable , Assistive Smartphone Peripheral for
What could be done to improve the app?<br>
Reading Text. MASSACHUSETTS INSTITUTE OF TECHNOLOGY June.</ref><br>
He would like a function added where the teacher can fill in a function and it is displayed on the screen in the same way as the levels.
An article about a device called the MobiReader, a mobile device to assist blind people in reading. It is a camera and text to speech technology. While we aim for text to braille this article can help with the input of our device. The camera conversion system is something we could find helpful.


= Planning =
The ideal test case went very positively. Two main things could be concluded from this test. The menu navigation was too difficult and should be improved or changed. The other conclusion is that the level itself worked really well and helped visualizing the shape of a function.


Below is a shown the planning. If a cell is colored green, it means that the task presented in the same row should be done in the week of the same column. In red are shown the milestones.
The less ideal situation was also tested with international students between age 18 and 21. They were blindfolded to represent a blind user. They were all able to complete the levels and understood what shapes and functions they just traced. THey all thought the app would be helpful and intuitive for blind users. The best feedback we received from this group was that the graphs should be made wider, making it less likely to slip of the graph. This way it would be easier to understand the shape that was just followed.


[[File:PlanningGroup2q1.PNG]]
= Conclusion =


= Research =
Our project's idea was to create an application/prototype targeted at mainly blind and visually impaired users, as well as sighted people. The reason we decided to make it usable for sighted users as well was that after doing research on the success of application and what makes an app appealing for our main target group (blind users) we found out that in order for an application to be picked up, successful and relevant for a long time it also needs to be appealing to sighted users as well.


== Scanning technique ==
Our application purpose is to teach users about math and more specifically geometry (an aspect that visually impaired people can study efficiently) in a more entertaining and interactive way. We first designed a theoretical concept for the app containing all requirements and features the final app should have. Then we made a prototype version of this theoretical app.


The device must have an effective way to scan or to photograph documents in order to be further processed. The images taken have to fulfill certain requirements if the OCR has to recognize the characters. A scan or photograph of a very high resolution would result in good recognition and an reliable conversion but it would also result in low processing speeds. It would also require a very good memory card to store the information. On the other hand if the scan or photograph is to low in resolution the OCR wont recognize the characters. It was found that the minimal amount of dpi (dots per inch) should allow the smallest feature, such as the dot on the i, to still contain at least 4 dots. This fact can be used to find the camera or scanner requirements but more on the way to scan has to be found. A lot of scanners have a lot of movable parts. These parts are very vulnerable if the device is shaken to much. This brought up the idea to not have something internally move, but allow the user to move the device to scan the paper. After some internet searches this type of technology proved widely available.
The application prototype is called "Learning Curve" and is made for Android OS platforms such as tablets and smartphones. It was done using Android Studios, and it uses a very unique style of interacting with the users.
It uses audio feed, vibration and screen tracing in order to present a shape/ figure on the screen (otherwise invisible on the completely black screen space). This method of interaction is at the core of "Learning Curve" because it provides entertaining exploration and is a challenge to the user. By challenging the user we try to make them more involved and interested in completing the task that would otherwise be mundane and boring.


== Interview with Visio ==
After creating the prototype we did a number of tests with different test groups:
The main test group was kids with visual impairments and blind children from the Center for blind and visually impaired in the town of Grave, with which we had correspondence throughout our project.
Two other test cases were sighted people who volunteered to test the app and experts from the Center in Grave. The overall feedback we received was positive with the users liking the concept and usage. There weren't many negative comments about the application apart from suggestions on how to further improve it. The people were pleased with what we presented to them.
This made us conclude that our approach for such an application was successful.
In the end we were able to cover all our goals and set requirements as well as being overwhelmingly happy about the response we received from users.


We wanted to get a better impression of what our users would require and would appreciate from the sort of device we are aiming to make. We therefore contacted Visio, an organization which specializes in assisting blind or visually impaired people with performing their dialy tasks, and tries to make their lives as comfortable as possible. They provide advice and assistance to blind people, as well as develop new technologies which aim to imrpove the quality of life for blind people.<br>
The project was a unique experience that gave us a lot of insight on the covered topic as well as making us learn the intricacies and importance of successfully working in a team.


We sent an email to Visio expressing our interest in visiting their location in Eindhoven. We got a reply from Erik van Dijk, an employee of Visio, that it would be preferable for them if we first called. We did so, and what followed was a conversation of about 20 minutes in which Erik explained to us that what we were trying to do has already been done in quite a sophisticated manner, and is already being used by blind people, as well as being covered by insurance.<br>
= Discussion =


He named a couple of products which were not exactly like our idea, but did more or less covered its entire range of functions, while also providing extra applications which make this device a lot more likely to see practical use than our own. One of such devices is called Optalec EasyLink12 Touch (see https://nl.optelec.com/producten/el000003-
Having made a prototype of our main idea, we can now discuss the impact, usefulness and reliability of our findings from this prototype, and connect this to the complete picture of the game, as listed under objective 1. For the prototype, four requirements were set, the last three of which have been met unquestionably. However, the first requirement, “The app needs to be navigable without any visual information” is perhaps not so easily met. The user interface as it is contains relatively large buttons, with equal spacing between them. A single click on a button will result in having the phone read out loud the information readable on the button, and a second click will result in performing the action listed on the button. From the user test case, it turned out that this alone was not enough to have the game be instantly accessible to blind users. The reason for this being that the user would first need to localize all buttons before they can tap a single button which does what the user wants the game to do. This could be fixed by applying the same accessibility features that are used on devices that run iOS. Here, the user can swipe their finger over the screen, and every time the fingers hovers over a piece of text, this is automatically read out loud. This method is also preferable because it uses the same mechanisms that blind users already use to operate their phones.<br>
easylink12-touch.html), and can be fully integrated with a smartphone. It can present text from a phone in braille, or it can read it out loud depending on the preferences of the user. If someone were to take a picture of a piece of text with their phone, use OCR software on to convert the text in the image to characters that the phone can recognize, this device can do exactly the same as our own device.<br>


Erik went on further to explain to us the current challenges in developing technology for blind people. These included videogames for blind children and exploiting the possibilities of smartphones to increase the ease with which blind people could go shopping in a supermarket, as this, explained Erik, is currently quite a nightmare for them. He mentioned SeeingAI, VisionAI, iFarkle and Amazon Go as current points of interest.<br>
The app itself, or rather the prototype thereof, is as of yet not without its bugs. After a period of time using the app, the sound will disappear from the app. This can be easily fixed by restarting it, yet it remains an inconvenience, especially for those reliant on sound for their navigation. It is obvious this needs to be fixed, but unfortunately, we didn’t have the time for that. <br>


Following this interview, we decided to drop our current idea. The chances that we could make any significant improvement to the device that is currently used are slim, since it is already highly advanced and would require a group of specialists to make small improvements.
Now, despite these imperfections in the prototype, the game itself works exceedingly well. Both the sighted users and visually impaired user on which we tested this app were able to deduce a function from a vibrational representation. Besides this, the enthusiasm of our test users shows that there is real potential in this form of a non-visual math game. Don mentioned that it would be impossible to make any definite statements on the usefulness of such a game in an educative environment right now, and that it would need to be tested for about a year before such statements can be made. With this we would agree. <br>


Of course, should the game concept turn out to be unpopular among our user group, the software that we has been developed for the prototype can also be used as a quick way of generating a tactile representation of a graph. If a device with a sufficiently large screen and vibrating function can be found, the application of our current software to such a device could be a valuable addition in teaching math to visually impaired students. If the software were to be expanded such that it becomes possible to modify a function, for instance add a vertical displacement, and have the device then instantly display the modified function, it would accelerate the current rate at which tactile graphs can be produced enormously, provided that a representation using a touch device is equally informative as a printed graph.<br>


= Solution =
It would appear that there is much to expand upon in our prototype, and that it bears a lot of potential. How then for our much broader concept of a math game, as listed under objective 1? Unfortunately, it is difficult to say concrete things about this, other than that extensive testing is required to make sure that how we envision the game to be played is clear for the user, and that the different game modes are experienced as being enjoyable and not a chore. The section “Features” under “Objective 1” gives an elaborate explanation of the setup, and there doesn’t seem to be anything else that needs to be done, except implementing everything listed into the actual app.
The core of the concept consists of the following subsystems:


* A scanner that directly contacts the surface which contains the text to be scanned.
= Afterword =
* A small (single-board) computer that can run OCR software and control the physical braille pins.
* A refresh-able array of braille pins to present the scanned text to the user.


A first sketch of the device can be seen in figure 1 and figure 2.<br>
Over the course of the project we had many exciting experiences and grew as a team. We managed to do our task with 2 weeks less than normal due to our change of idea. We learned a lot about how group work should be done and that working on a project is far more time demanding and complicated than it sounds. Nevertheless we managed to keep our enthusiasm and interest in the topic throughout the whole course.


<div style="display: inline; width: 1300px; float: right;">
A moment that that inspired us all is when we got the chance to visit the center for visually impaired children in the town of Grave. It was an opportunity for which we are very grateful and it was all possible with the help of Don van Dijk, who was very enthusiastic about us making a project related to his work and passion - helping blind and visually impaired people. Going to the educational center gave us a first look in the lives of visually impaired children and how brave, optimistic and ambitious they are. Without a doubt that single meeting we had motivated us to not only work on the project because of the course but also because we believed our idea is a just one and that it can lead to future improvements and innovations that would benefit those people.
[[File:Braille_V1_A.png|600px|thumb|left|Figure 1 A first sketch of the concept showing two buttons on the bottom of the device to scroll through the text.]]
[[File:Braille_V1_B.png|600px|thumb|right|Figure 2 A first sketch of the concept showing an on/off button on the top of the device.]]
</div>
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>


There are two dark grey buttons on the bottom on the device used to flip through the pages of the scanned text. This is necessary since not very much text can be displayed on the screen at once and it also allows one to re-read some text that was scanned already. The top of the device contains a on/off button for the device. The text displayed on the device in figure 1 and figure 2 is actually the problem statement of this wiki in braille.
The process of coming up with the idea for our project and the realization of it was a long and with many obstacles along the way: from lack of desired hardware to the need of research and the desire we had to come up with something that is unique and innovative. We all learned new an important things during the course of the project, from teamwork, organization and time management to working with Android Studios and the information and knowledge needed to make an application for a user group that is so unique and foreign to the smartphone game sector. The feedback we got from our test cases tells is that people find our idea interesting and helpful. In our eyes it is not just a university project but also an aspect of education for visually impaired that can be further improved. It is something non of us will forget and we might work on it further if the situation and opportunity arises.  


= Discussion =
In the end we are all pleased with the journey we went on and the outcome of it. We are satisfied with the work we have done and the lessons we have learned, which will without a doubt be useful in our future.


= Conclusion =  
= References =


=References=
<references />
{{Reflist}}


[[Coaching Questions Group 2]]
[[Coaching Questions Group 2]]

Latest revision as of 23:34, 5 April 2018

0LAUK0: Robots Everywhere group 2

Group members

  • Yngwie Baron (0936539)
  • Axel Deenen (0947031)
  • Moos Müller (0936214)
  • Dimitar Nikolov (1000095)
  • Wybe van Vlokhoven (0914565)

Project definition

Problem statement

There are not many resources or convenient ways to practice math and mathematics related skills for blind and visually impaired people. Solving a mathematical problem is largely dependent on the ability to visualize the problem. Without a proper way to visually represent math and observe the whole exercise it becomes very tedious to do problem solving effectively. Take for example exercises on geometry or graphs, the lack of visual aid makes these tasks much harder and sometimes close to impossible as it requires you to remember information you would otherwise be able to look back at. Current methods used to teach blind people are time consuming and expensive, according to our expert Don van Dijk from Visio. These reasons make it less appealing to young blind or visually impaired children to learn math and math related sciences. This leads to an unfamiliarity with math among them while a good mathematical basis is important for everyone.

Besides the lack of convenient ways to teach math, there is also a lack of games for blind children that they can play. Most apps and games are made solely for sighted people with lots of visual cues. The blind do however like the few games that they can play very much and would like there to be more available games for them. In this project a solution for both these problems is created.

Approach

The way the final product is made consists of several stages. In the first stage knowledge is gathered. During this stage sources will be found on fields of educational benefits of games, games for blind people but also the way mathematics is taught to the blind. Interviews with an expert on the field of educating blind people will give insight into the possible requirements for the device. During this stage the things that the device should be able to do are defined. After all the requirements are quantified and the desired abilities are listed the second stage can begin. The second stage will be focused on writing the software necessary to fit the requirements. The final product would be an app that will run on a smartphone. During this stage it is checked if all the requirements on what we aim for the end of the course are met and everything works. Below a link to all the sources and the planning can be found.

Sources and planning

State of the art

Figure 1: Tactile printer with an example.
Figure 2: Example of a 3D-printed graph.

Sources

The state of the art was first checked by acquiring 25 sources. These sources were about a couple main topics, e.g. teaching mathematics to blind children, educational games, Braille and apps for the blind. This research gave a better understanding of what was important to take into account when making an app, for example voice control on a phone, but mainly the struggles with teaching math to people that are not able to visualize graphs and shapes. The articles also described some of the solutions and devices that were already implemented currently. However, the articles were mainly about the system in the USA and since the project was based in the Netherlands, further research needed to be done. This further research was also useful in another aspect. Some technologies and devices were read about in the sources, but how they were used in practice in schools was still very unclear.

Visio

Therefore we contacted Visio, a Dutch company that works with and for the blind and visually impaired. Through this company we met one of their employees, Don van Dijk, who has experience with teaching math to blind children. Don gave a lot of useful information. He made clear that the biggest problem in the education of math to the blind is currently that graphs and images are really hard to depict in an understandable way. The company Dedicon can make tangible drawings for them on request, but this takes a long time and could be improved. He also told us that the the Nemeth code is not used in the Netherlands, because another system is developed. Therefore there was no need to use or teach this in the app. Don also gave some useful sites that already had good accessibility for blind users. The best example was desmos.com, a graphical calculator that uses a lot of features for blind users. On this side they use audio trace to visualize a graph, examples can be found here.

To get more information about the current systems on schools for the blind and visually impaired, Don offered us a visit to Visio's school in Grave. The tour was given by Don van Dijk himself. It was a very educational excursion where we were shown all the devices available to them and their use in practice. A student taught us how he operated a computer using the 'Brailleregel' for example. Besides this Don explained to us that they used some really old devices that still worked well for them, e.g. a Braille typewriter, and also some newer once.

One of the devices that worked really well for them was a tactile printer, a device that helped them explain graphs to their students. Through heating a special piece of paper the black ink on it could create a ridge on the paper. This printer with an example can be seen in figure 1. According to Don this system was most intuitive to the blind and would best help them visualize a graph. Closely followed by a 3D-printed graph, see figure 2. Besides these two ways to some other aids that they used were pin boards, tangible shapes and shapes and graphs created with wire. [1]

Don was also interested in the idea of making a game. He explained to us that the games that would last and work best were the games that were made for more users than just the blind. It should have either a function for the blind to be able to play it or not depend on sight at all. The game that was currently popular at their school was a game for iOS called BlackBox. The goal of this game is to complete all levels, and each level is a puzzle the user needs to solve. Little to no hints are given to the user what the puzzle actually is about, meaning that the hardest part is figuring out what needs to be done. This element which made the game unpredictable seemed to be the sort of challenge that the children there could appreciate.

USE analysis

The USE analysis will cover how the app will affect the Users, Society as a whole and Enterprises as well. every target group will be addressed separately.

User

The primary users for our application are blind or visually impaired children between 6-16 years old. The reason for this is the application topic is geometry and it is at that age that kids start learning about this topic. The fact that humans learn things a lot faster, easier with a more long lasting effect when they are young as opposed when they are adults also plays a big role. Another factor is that the application has the appearance of a mobile game (similar to "Brain quiz" games on the market), therefore being appealing to kids more so than adults.
That is not to say that adults can't play the game. Adults with visual impairments or blind people can also effectively play the game as it is a unique way of learning about shapes and geometry. They still have the "interest factor" since this is something a lot of blind and visually impaired people don't every get the chance to learn (learning geometry in braille is a very complex, tedious and off-putting to many people). Since this application does not involve any braille it is an alternate way for them to delve in geometry.
And of course "Learning Curve", which is the name of our game, is perfectly suitable for use by sighted people. In this case it still retains the key aspect, where the user has to "sense/feel" the shape or curve on the screen without seeing it while having the menu perfectly visible. Due to the uniqueness of the game it would strike interest in them.
Last but not least teachers that teach to blind and visually impaired children can use the application by being secondary users. They can try to incorporate it in the teaching of geometry if it shows a good response from the students. This will benefit the teachers indirectly as it will reduce the workload they have and make their job easier and with potentially higher quality in terms of results.

Society

The application benefits Society by closing the gap between blind/visually impaired people and people with eyesight. In our modern time a lot of things are developing at rapid speeds but one that always lacks behind, because of its complexity, is the societal imbalance between those two groups of people. Because of their inability to see, blind people do not have access or the means to enjoy many of todays technologies, qualities of life and other aspects, all ranging from basic activities to career and normal social dynamics. While the integration of blind people in modern society has been improving every year, a lot of aspects are hard to overcome. The "Learning Curve" application has the potential to assist in teaching visually impaired people in learning about shapes geometry and therefore reduce the societal gap between sighted and blind people, that occurs on an everyday basis. The imbalance in accessible knowledge between the two groups leads to visually impaired people being unable to be members of certain social sectors (professions that require geometry knowledge like for example). Since our application provides a unique method to learn about geometry we believe it can be a great benefit to society.

Enterprise

Since "Learning Curve" is a non-profit free application it provides no direct benefit to the enterprises. It can, however, bring more light on the topic of "Applications for blind and visually impaired", if it gains popularity. This will of course bring competition, as is the case with most innovations. A new fresh market would mean that new competitors will join, wanting to get a stake in it. This is not only beneficial to the companies and developers themselves, but it is also going to lead to further innovations and advances in the sector of "Applications for blind and visually impaired".

Research

The effectiveness of educative games

The effectiveness of educative games is largely based on two factors: the educational value of the game and the attitude of students towards the game.

Educational value

The educational value of a game is based on how much knowledge is packed into the game and the skills that might be taught by playing the game. “What are the students going to learn from the game?” Is an important question to answer. The game should teach topics which are not easily taught in another form and the time spend on playing the game should be proportional to how much the students learn. It is important to notice that the learning content not only consists of the knowledge taught, several skills are also developed. For instance, gaming could improve problem solving, collaboration, communication and social skills.

Students’ attitude

Students have varying types of personalities and therefore all respond differently towards the use of educational games. The seemingly positive effects of games may not be the same for every student. The attitude of the students towards the games is a deciding factor in their behavior. A study of the attitude of students towards an educational game is crucial to its success. A student’s attitude is influenced by four factors: Relevance, Confidence, Media affinity and Self-efficacy[2]. These four factors will now be explained in more detail.

The relevance of a game should be clear to the student. The relevance is related to the content learned and to the way in which it is taught. If, for instance, a game has a high educational content but the way it is taught is very awkward the perceived relevance is low. On the other hand, if the game is very well designed but it has little to no educational content it would no longer qualify as an educational game. A student should believe the content learned with the game is best learned using the game. The game developed in this context might have relevance to the student because it will help them visualize a graph or shape in different ways. By using various sensors in their phones this way of visualization may only be achieved by the described game.

The students’ confidence in succeeding will influence the students’ persistence. They should not be worried about their inability to properly use a game to learn. Frequent feedback in the form of rewards during the game will increase their confidence and various game levels (easy, medium, hard) will allow the student to learn at their own pace. The graph game will increase the user’s confidence by giving of both visual as audible rewards after completing a level. It will also remember the amount of levels that were completed to stimulate a sense of achievement.

Media affinity is the importance that a medium has in the lives of the user. Research suggested that, for example, the affinity towards the use of a mobile phone had a positive influence on mobile shopping[2]. The iPhone and other Apple products are of major importance to visually impaired and blind people, as was told to us by Don van Dijk. These products help them to function as normally as possible by the use of several features such as voice assist, resulting in a high media affinity.

Self-efficacy is the belief of an individual in their own abilities to achieve a desired outcome. The student has to believe he/she has the ability to finish the game and to gain the desired knowledge to play the game. Self-efficacy might, in this case, be influenced by eliminating the possibility of failure and letting the student get acquainted to the game by the use of a simple tutorial.

Considerations on entertainment

As explained in the section “on the effectiveness of educative games”, the amount of new things that a student learns from an educative game should be proportional to the time spent playing the game. Clearly then, it is desirable that the game is played extensively by the student.
This desire can be fulfilled by exploiting one property of games: their goal is not only to educate, but also to entertain. If this last characteristic is implemented properly, the students will voluntarily occupy themselves with the game.

Mechanisms of entertainment

The question then becomes: how can an educational game be made entertaining, without comprising its educational core? Perrotta et al[3] have done extensive research into videogame based learning, and have listed a number of principles which should be ingrained in the design of educational games, as well as the mechanisms by which they can be implemented in a game. Below are listed the mechanisms which can provide entertaining value.

Challenging goals

Providing a clear goal will show the users the result of their effort. It should be clear towards what the users are working. The goals should be quite challenging, or else the users won’t take the game seriously or they will get bored.

Our game could implement this by having a number of levels, each pertaining to a different function and corresponding graph. If the game is too easy for a user, he or she can advance to levels using more difficult functions.

A fictional setting to provide a compelling background

Perrotta notes that this feature should not be used as a means of escapism for the users, but should be used to provide them with an environment in which they feel free to experiment with different methods of approaching a problem, without risking actual failure as in real life.

Difficulty levels

Using different difficulty levels through which the users can advance progressively will stimulate and challenge the users to improve their performance. Again, the criteria for advancing or “leveling-up” should be made explicit. Advancing to a next level can be combined with a sort of reward, thereby acknowledging the user’s mastery of the material. This also increases the feeling for the users that they are in control, and are directly responsible for their performance.

Uncertainty

Adding a non-linear element, such as the ability for the users to choose different tasks themselves instead of always being presented with a task, also increases the user’s feeling of control. Doing so will encourage the user’s to think independently and with that, increase their capability of learning new things.

Users should be able to select themselves different levels in our game. They won’t have preliminary knowledge about the actual contents of a level, but it would be possible for them to know something about the relative difficulty of a given level.

A social element which allows the users to share experiences

Another advantage of games is they allow for a strong social experience. If users are encouraged to share their findings and knowledge about the game so that others can advance, users may form bonds. This will increase the enjoyment users experience from the game, besides allowing them to develop additional social skills.

An online ranking system could be added, so that users can compare their performance with their friends.

Possible ways a phone can be used to sense a graph

If our game is to be in the form of an app, we would like to use the full input potential of current smartphones. Below is a list of different ways users could perform different tasks, with respect to a game involving graphs.

Line to formula

  • The user traces the graph by moving his/her finger along the line. If the user’s finger diverts to much from the line a vibration will be produced. The further away once finger from the line the harder the phone will vibrate or the shorter the time between vibrations.
  • The user follows the x-axis and the higher the y-value the higher the pitch of the sound that is being produced (DESMOS). Or the user traces the graph and the further away the finger is the higher the pitch.

The levels below have to be performed in a controlled environment. The user gets the formula of a graph through audio and visual signals. It uses a combination of all the sensors in a phone.

Formula to line

  • The user has to move the phone according to the shape of the graph.
  • The GPS is used to calculate the speed at which the user is moving. This allows the user to move in a straight line while accelerating and decelerating according to the shape of a graph. A good way to learn derivatives.
  • The user has to move in the shape of the graph. The movement is traced by the GPS.
  • The user draws a graph on a piece of paper and takes a photograph of it. Recognition software will check if the right graph is drawn.
  • The user sings through the microphone at the right pitch to represent the height of the graph.

Description of the educative game

The game consists of a collection of levels, which do not necessarily have to be completed in order. The ultimate goal of the game is to complete all the levels. By completing levels the users gains points which the user can trade for hints, new levels or new sounds. There are two game modes that will be imlemented in the game. The goal of the first game mode is to figure out what function it is that a level contains by tracing the graph of that function. The goal of the other mode is to draw a graph corresponding to a specified function. However, each level represents the graph or requires its input in a completely different way, but it is certain that in none of the levels the graph will be visual on the screen. The users will need to use various senses and methods to try to visualize the graph. They will also need mathematical intuition and knowledge to connect the shape they have in mind to a function, or the other way around.

The core learning part of the game is to gain “a feeling” for the behavior of functions and shapes. As stated, users will receive no information whatsoever on how to solve a level. This is much the same as with the app Blackbox, which was according to an expert from Visio a big success among the blind and visually impaired children (as well as seeing children). Since users do not even know how to describe the graph, they will have to use all possible ways of inputting information they can think of. It is this that will challenge the creativity of the user, and also be the most enjoyable part of the game. Herein lies also the biggest design challenge, since all levels need to be more or less unique to keep the game challenging and surprising.

The game will also contain an online ranking as to encourage users to play it with friends and help each other solve problems. This can be done as follows. When users complete levels, they can gain points. The amount of points they get will be proportional to their performance, measured by a number of parameters such as speed and accuracy. For example, users could get upon completion of a level either one, two or three points. If they complete it without mistakes, i.e. correctly following the shape of graph at all times, and within a certain time limit, they get full points. A point can be subtracted if between one and three mistakes are made, and only one point is awarded when more than three mistakes are made.

Users can see a ranking of their performance based on the total number of points they have, compared with other players. In addition, if a user completes a level with three stars, that user gains a point. This point can be, as described above, used to obtain ornaments such as new sounds, or it can be used to enable more functional features such as new levels or hints on how to complete an existing level.

In addition, users can assist their friends not only verbally, but also by donating them points, so their friends can unlock hints for difficult levels.

Solution

The Solution section is split in two parts the "Objective 1" and "Objective 2". Objective 1 elaborates how we envision the solution being done given we had enough time, knowledge and resources to work with, and "Objective 2" represents what we were able to achieve during the span of the course. Everything is written in such a way that this idea can be easily expanded and the full functionality of the app can be realized.

Objective 1: Create a concept of an app that teaches the behavior of graphs to blind children

Objectives

The End product idea that we envision is an app called "Learning curve", which engages blind children in a light-hearted manner with the visualization of graphs.
The app should provide children with a brief explanation of a mathematical function. This explanation can be read out loud by the app. This is to give the child the necessary conceptual knowledge. The core of the app will be a game in which children are challenged to apply their creativity and knowledge of functions to try to figure out which graph it is they need to identify. In short, we aim to make an app which turns what is ordinarily a tedious and slow process into something enjoyable for blind children.

Requirements

Note: These are the requirements that we envision for a proper end product version of our app, i.e. for objective 1.

  • A tutorial needs to be present that introduces and teaches various functions and their corresponding graphs.
  • A game which challenges the player to visualize functions and understand their shape.
  • The option to play with other players online.
  • Incentives to keep the player engaged with the game on a long term (gain points for playing and more if one does well, online ranking so one can compare himself/herself with other players).
  • Game needs various functions such as audio and vibration.
  • A practice mode that is single player and can be played offline.
  • Multiple difficulties where more complex functions are used on a higher difficulty.
  • Input from the user is obtained through various ways such as a touch display, microphone and accelerometer.
Figure 3: The structure of level selection.

Features

The end product is an iOS app that can be downloaded from the app store. As discussed earlier the final product should be made for Apple since most blind children use Apple products. There are a few features necessary to make the app both educational and enjoyable.

Figure 4: The structure of different functions in level selection.

Suggestions for the different levels to be implemented are described in the "Possible ways a phone can be used to sense a graph".

Since navigating through the app is important (think of choosing levels, using hints or changing settings) the menu's have to be specially designed to be accessible to blind, visually impaired and sighted users. Navigation through the app will make use of the 'voice over' function. This voice over function has the ability to read the text written on a button on the screen of the phone when the button is clicked. Only when the button is clicked again will it trigger the button. If, however, the user click anywhere else on the screen the button will reset and the user has to double click it for it to trigger. Everything clickable in the game will have sound, and buttons with similar functions, such as a button for going back to level select and a button for going back to the main menu will have similar sounds, but with different lengths and pitches. This will allow blind users to navigate the game easily, but the sounds also serve an important role in creating an enjoyable and immersive game enviroment[4]. Furthermore, the buttons will all be fairly big to ensure that clicking the right thing is easy and visually impaired users can see them better. The buttons will also contain written text. The main menu will contain five buttons: level selection, explanation, online ranking, shop and options.

The level selection button will take the user to the next screen in the app in which all the levels are displayed in the form of icons with numbers in them. When a level is clicked on it will not only state the name/number of the level but also if it has been completed already by the user, to help the user keep track of the progress made (of course the user is free to complete a level multiple times). When a level is selected the user will be send to a next screen which is build up to match the desired functions of the level. An example of a level could be a level in which the users has to move its phone through the air, trying to follow the shape of the unknown graph. The more the shape traced with the phone resembles the graph the longer and higher a pitch will sound. When the path traced through the air with the phone resembles the graph enough (correct amount of maxima and minima, correct qualitative increase or decrease in slope) the level is completed. It could take quite a while before the user figures out how to "see" the graph. Before the user knows the movement of the phone is the input, the user might experience moments in which the game notifies the users that the answer is wrong, while the user does not know why he receives that message. In this exploration aspect lies the fun of the game. Trying to think of functions to solve the level, while not having all possible obtainable information could also learn the user a lot about the graphs of functions. Only if the player has given the right answer in a level, the next level will be unlocked and the user is taken back to the level selection menu. Points will be reward after completing a level. Giving a wrong or right answer will trigger an appropriate sound.

Figure 3 shows the structure the level selection menu will have. A level starts off by tracing the graph on the phone, followed by answering a multiple choice question to check if the user understood what function he just traced. After answering the multiple choice question correctly the two 'drawing the graph' levels unlock. These drawing levels are two options from the list in the 'Possible ways a phone can be used to sense a graph' chapter. For example this can be drawing the graph on paper or having to walk the graph. To the user it is unknown which options the levels use and that is what he/she has to figure out. The options are different for every new graph he/she unlocks.

Besides the two drawing levels, after completing the multiple choice question, the next function also unlocks. This is shown in figure 4. The next graph unlocks after answering the multiple choice question correctly, because then the user can choose to continue to next graph if he/she does not want to or cannot complete the drawing levels at the time he unlocks them. In figure 4 it is shown clearly that the multiple choice levels and the drawing levels are on different paths. The further the user goes from the first graph level the harder the functions are. This increases the difficulty over time for the user when he is more familiar with the workings of the app and has a better visualization of the easier functions.

The explanation button gives the user the needed knowledge to complete levels. It will contain some general hints and possible levels they can encounter. This way they know what to expect from the levels and the game as a whole.

An online ranking button will be added to give the users the drive to compete. The button will take the user to a list with other users. The list can contain global users, national users or a closed group of users (think of a list of classmates). After opening the online ranking the voice over will tell the user their current rank. Hereafter, the entire list will be read out. On the screen below there will be a skip button which, after clicked, skips a desired amount of people (this can be set in the settings).

The points scored can be used in the shop. This shop will contain the possibility to purchase sounds, different colors or even levels with the point system. Spending points will not effect the ranking of the player.

In the options menu the settings of the game can be adapted. The voice over function can be toggled on and off, a high contrast mode can be selected and the online ranking settings can be adapted. Also the language can be changed here. Some basic colors are already available here to adjust to different types of visually impairment. This way they can adjust the colors to the once that makes it the easiest for them to see.





Objective 2: The achieved game

Requirements

Due to lack of time, the actual app will not fulfill all the requirements listed under objective 1. Instead, we aim to create an app which incorporates some key elements of the above mentioned properties. This will allow for a proof of concept, i.e. see to what extent an app focussing on learning the behavior of functions will be helpful for and appreciated by blind children. The requirements that are needed for such research are listed below.

  • The app needs to be navigable without any visual information.
  • The app must contain a visual interface, so that it will be accessible to sighted people as well.
  • The app needs to contain one fully functional level, including the depiction of a specific function, and the option to answer which function it is. This also encloses the possibility of giving a wrong or right answer, and the app needs to indicate whether a wrong or correct answer is given.
  • The app must have a general user interface, which includes a main menu from where the user can choose to play a level.

Deliverables

Since time is limited the goal of our project is to create an Android game app as described above containing at least one working level. Our initial idea for a solution to the problem was a game in the form of an iOS app. This platform, instead of Android for example, is chosen because it was found that the majority of blind people own an iPhone due to it having very user friendly capabilities for blind users. However, after some delibration, it was decided that the prototype of our app will be made for Android. The reason for this is purely practical, in the sense that none of the group members possesses or has any means of acquiring a device which runs iOS. This doesn't detract from the value of our project, since we aim only to build a single level to test the feasability of an educative math game which is meant specifically for gaining intuition for the shape and general behavior of graphs. Also we believe that the platform is of little importance as we can achieve the desired effects on Android devices as well. All this will be discussed in further detail below. The game is also designed such that it can be played both by seeing, visually impaired and blind people. This is done to create broad support for the app, as advised by an expert from Visio. The app to be delivered does not contain all the features described in objective 1. It will not contain the point system, the shop, the online ranking and an options menu. Nor will it contain the settings or a wide variety of different levels. The code is, however, written in such a way that it can easily be extended to fulfill all the features.

The game will only have one gamemode, namely to recognize the graph and answer the multiple choice question. Therefore the level select screen contains only a grid of levels instead of a distinction between the different game modes. The game will also contain only one level type, as in that the vibrating function of the phone will be used, in all three levels, the indicate the shape of the graph. But this does not result in a change in the level select screen compared to the theoretical concept of the app.

The developed app

The logo of our app "Learning Curve".

Main menu

The first screen the user will encounter when opening the app is the main menu as seen in the screenshot in figure 5. The main menu contains five large buttons labeled: “level selectie”, “uitleg”, “online ranking”, “winkel” & “opties”. When one of these buttons is tapped the text on the button will be read out loud by the phone. Only when a button is clicked twice without clicking on another button in between will it perform an action and play a clicking sound similar to that of a chest being opened. The only button in the main menu that leads to another screen is “level select” and because of this it is the only button that will play the clinking sound when clicked twice in a row. Since the app is meant primarily for blind dutch schoolchildren, the interface is in Dutch.

Level selection

The level select screen contains twelve square buttons in a grid all labeled with a level number as seen in figure 6. When first opening the app only level 1 will be the same bright blue as the rest of the app to indicate to the sighted users that this is the only unlocked level. When level 1-3 are clicked once the number of the level (as well as the word level) will be read out loud. When the user clicks again on level 2-3 the app will tell the player that this level has not been unlocked yet. When the user clicks again on level 1 it will enter level 1 and play the clicking sound. If the user clicks on level 4-12 the app will tell the user that “this level is not available in the demo”. Only level 1-3 are functional and each time a user completes a level the text of the next one will be turned to the bright blue and it will be unlocked as can be seen in figure 7.

The level

Again the goal is to find the function belonging certain graph, where the graph can be represented through various ways which are unknown to the player. In this level the graph will be communicated to the player through the touchscreen and vibration function of the phone. The player can move its finger across the screen of the phone and if the player is within a certain distance to the graph then the phone will vibrate. Otherwise it will not. Now that the player can trace the curve he can visualize the function. When ready the player can click on a button in the bottom of the screen. When clicked the player can choose from 4 multiple-choice answers which will be located in the four vertical quadrants of the screen of the phone. The answer will be read out loud when clicked on, and will be chosen when clicked on again (as is with all buttons). When the player chooses the wrong answer the level needs to be solved again, but with a different graph. It is to be noted that in our case we only have one graph hence selecting a wrong answer will not result in a new graph rather the same one. The level screen contains only one button named “antwoorden” as seen in figure 8. This button is to be clicked (again twice) by the player when he/she thinks to know the answer. When the user touches the (black) part above the button of the screen the phone will vibrate unless the finger of the user is within a certain (shortest) distance to the graph. The user will need to figure this mechanic out for themselves, as it is part of the fun exploration in the game the blind children enjoyed. The app contains three working levels with the functions y = 0, y = x & y = x^2 (in that order).

Answering

The answering screen contains four button labelled each with a different function as seen in figure 9. Again the voice-over double click function is implemented. When the user clicks the wrong answer a loud buzzer will sound indicating that it was wrong and the user will be redirected back to the level the user was trying to answer so he/she can try to visualize the graph better. When the correct answer is selected a cheerful harp will sound and the user will be redirected back to the main menu.

fig5
Figure 5: The main menu screen.
fig6
Figure 6: The level selection screen when first opening the app.
fig7
Figure 7: The level selection screen when level 1 has been completed.
fig8
Figure 8: The level screen.
fig9
Figure 9: The answering screen.

Explanation of the code

The code works by defining two arrays corresponding to the x- and y-values of the graph. The number of elements in the x- and y- array are both equal to the width of the screen in pixels. The standard coordinate system on the phone (which has the origin in the left corner, the positive y-axis pointing downward and the positive x-axis pointing to the right) is converted to a coordinate system with the origin in the center of the screen. All the x-array elements are filled in in such a way that the first element corresponds to minus half the width of the screen and the last element is plus half the width of the screen. These x values are then used to the define the y values in the y-array according to a desired function.

The position of the user’s finger is then measured. A circle can now be drawn around this position with its radius corresponding to a desired margin. If the graph is not within this circle and the user moves his/her finger, the telephone vibrates. The vibration last for 5 seconds unless something else happens that will automatically terminate the vibration (i.e. if the user lifts his/her finger). The vibration will not restart every cycle because the program remembers whether it is on or not. This is achieved by the use of an array consisting of two elements, one for the current loop and one for the loop that was done before. If there is no change between the current loop and the loop before it, there will be no change. If they differ the vibration will be started or cancelled accordingly.

App download

Please feel free to test our android app on your own phone! Below is a .zip file containing the .apk file that you can install on your android phone. Do not forget to turn on “Allow installation of apps from unknown sources” in the security tab within the settings of your phone. And please note that the app might not function 100% correctly on every phone, since it only was tested on a couple of different phones.

File:Learning Curve.zip

Also feel free to improve on our app or have a look at the code in the zipped Android Studio project folder below.

File:Learning Curve Android Studio Project.zip

User test case

Because we did not know if the user test could take place before the deadline of the project, two test cases were made. One preferable situation were the test was done with a blind student and a teacher and a back-up test case done with blindfolded volunteers.

Ideal situation

The game is tested with blind children at the Visio Onderwijs Grave. A phone is given to the children with the app on it. The app is started already, since it is made on an android device and not on iOS. Because of this the app list cannot be read aloud. After the app is started the children should be able to operate the app without any assistance and be able to answer the questions correctly. Afterwards a few questions are asked. They are as follows:
Is the app intuitive?
What is the shape you just followed?
Do you think this can help you learn math functions?
Do you have tips on how to improve the app?

The following questions are asked to the teacher.
Do you think this apps helps the students?
Could you see this being used to assist teachers?
Do you think the app is useful?
What could be done to improve the app?
Do you have tips on how to improve the app?

Less ideal situation

In case the testing with the blind children is impossible another user test case should be held. In this case testing should be done on blindfolded volunteers with no understanding of the workings of the app. Again a phone should be given with the app launched, but besides stating that the goals aims to teach math, no instructions should be given. The blindfolded test subjects should be able to operate the app on their own. The only assistance allowed is translations for non-dutch speaking subjects. After the use of the app some questions should be asked.
Is the app intuitive?
What was the shape you just followed?
Do you think this can help visualize a function for blind people?
Was the lack of sight during the test an issue?
Do you think this can help teach math to blind people?
Do you have tips on how to improve the app?

Results

The test of the ideal situation was done with a 16 year old blind student named Tom and Don van Dijk. They both liked the idea of the app a lot and were very enthusiastic. Don found it however regrettable that there would come no final product from this project, because the idea and demo were interesting to him. The test case did have some issues with it and could not be done completely independent. The navigation of the menus was harder than imagined, therefore Tom needed help selecting the levels. The level itself was played completely independent. The difficulty of the navigation came with locating the buttons. There was no good solution for this implemented into the app. The text-to-speech helped him, but finding the buttons was hard.
These were the answers Tom gave us after playing the levels:
(Note: these answers are reproductions of his answers and not direct quotes.)
Is the app intuitive?
Yes. The navigation of the menu less so, because it was different than what he was used to. The level itself felt however very intuitive.

What is the shape you just followed?
In order a line, a slope and a parabola.

Do you think this can help you or younger students learn math functions?
Yes, he understood what shapes he followed and could visualize them.

Do you have tips on how to improve the app?
No.

The following questions are asked to Don.
(Note: these answers are reproductions of his answers and not direct quotes.)
Do you think this apps helps the students?
To give a definitive answer the app should be tested further and for a longer time in a class. After this it can be clearly said if the app helped or not. First impression of the demo is however positive and he could see it being helpful. Every way to visualize graphs to the blind is helpful.

Could you see this being used to assist teachers?
Yes, after further testing and development. See previous answer.

Do you think the app is useful?
It could be useful.

What could be done to improve the app?
He would like a function added where the teacher can fill in a function and it is displayed on the screen in the same way as the levels.

The ideal test case went very positively. Two main things could be concluded from this test. The menu navigation was too difficult and should be improved or changed. The other conclusion is that the level itself worked really well and helped visualizing the shape of a function.

The less ideal situation was also tested with international students between age 18 and 21. They were blindfolded to represent a blind user. They were all able to complete the levels and understood what shapes and functions they just traced. THey all thought the app would be helpful and intuitive for blind users. The best feedback we received from this group was that the graphs should be made wider, making it less likely to slip of the graph. This way it would be easier to understand the shape that was just followed.

Conclusion

Our project's idea was to create an application/prototype targeted at mainly blind and visually impaired users, as well as sighted people. The reason we decided to make it usable for sighted users as well was that after doing research on the success of application and what makes an app appealing for our main target group (blind users) we found out that in order for an application to be picked up, successful and relevant for a long time it also needs to be appealing to sighted users as well.

Our application purpose is to teach users about math and more specifically geometry (an aspect that visually impaired people can study efficiently) in a more entertaining and interactive way. We first designed a theoretical concept for the app containing all requirements and features the final app should have. Then we made a prototype version of this theoretical app.

The application prototype is called "Learning Curve" and is made for Android OS platforms such as tablets and smartphones. It was done using Android Studios, and it uses a very unique style of interacting with the users. It uses audio feed, vibration and screen tracing in order to present a shape/ figure on the screen (otherwise invisible on the completely black screen space). This method of interaction is at the core of "Learning Curve" because it provides entertaining exploration and is a challenge to the user. By challenging the user we try to make them more involved and interested in completing the task that would otherwise be mundane and boring.

After creating the prototype we did a number of tests with different test groups: The main test group was kids with visual impairments and blind children from the Center for blind and visually impaired in the town of Grave, with which we had correspondence throughout our project. Two other test cases were sighted people who volunteered to test the app and experts from the Center in Grave. The overall feedback we received was positive with the users liking the concept and usage. There weren't many negative comments about the application apart from suggestions on how to further improve it. The people were pleased with what we presented to them. This made us conclude that our approach for such an application was successful. In the end we were able to cover all our goals and set requirements as well as being overwhelmingly happy about the response we received from users.

The project was a unique experience that gave us a lot of insight on the covered topic as well as making us learn the intricacies and importance of successfully working in a team.

Discussion

Having made a prototype of our main idea, we can now discuss the impact, usefulness and reliability of our findings from this prototype, and connect this to the complete picture of the game, as listed under objective 1. For the prototype, four requirements were set, the last three of which have been met unquestionably. However, the first requirement, “The app needs to be navigable without any visual information” is perhaps not so easily met. The user interface as it is contains relatively large buttons, with equal spacing between them. A single click on a button will result in having the phone read out loud the information readable on the button, and a second click will result in performing the action listed on the button. From the user test case, it turned out that this alone was not enough to have the game be instantly accessible to blind users. The reason for this being that the user would first need to localize all buttons before they can tap a single button which does what the user wants the game to do. This could be fixed by applying the same accessibility features that are used on devices that run iOS. Here, the user can swipe their finger over the screen, and every time the fingers hovers over a piece of text, this is automatically read out loud. This method is also preferable because it uses the same mechanisms that blind users already use to operate their phones.

The app itself, or rather the prototype thereof, is as of yet not without its bugs. After a period of time using the app, the sound will disappear from the app. This can be easily fixed by restarting it, yet it remains an inconvenience, especially for those reliant on sound for their navigation. It is obvious this needs to be fixed, but unfortunately, we didn’t have the time for that.

Now, despite these imperfections in the prototype, the game itself works exceedingly well. Both the sighted users and visually impaired user on which we tested this app were able to deduce a function from a vibrational representation. Besides this, the enthusiasm of our test users shows that there is real potential in this form of a non-visual math game. Don mentioned that it would be impossible to make any definite statements on the usefulness of such a game in an educative environment right now, and that it would need to be tested for about a year before such statements can be made. With this we would agree.

Of course, should the game concept turn out to be unpopular among our user group, the software that we has been developed for the prototype can also be used as a quick way of generating a tactile representation of a graph. If a device with a sufficiently large screen and vibrating function can be found, the application of our current software to such a device could be a valuable addition in teaching math to visually impaired students. If the software were to be expanded such that it becomes possible to modify a function, for instance add a vertical displacement, and have the device then instantly display the modified function, it would accelerate the current rate at which tactile graphs can be produced enormously, provided that a representation using a touch device is equally informative as a printed graph.

It would appear that there is much to expand upon in our prototype, and that it bears a lot of potential. How then for our much broader concept of a math game, as listed under objective 1? Unfortunately, it is difficult to say concrete things about this, other than that extensive testing is required to make sure that how we envision the game to be played is clear for the user, and that the different game modes are experienced as being enjoyable and not a chore. The section “Features” under “Objective 1” gives an elaborate explanation of the setup, and there doesn’t seem to be anything else that needs to be done, except implementing everything listed into the actual app.

Afterword

Over the course of the project we had many exciting experiences and grew as a team. We managed to do our task with 2 weeks less than normal due to our change of idea. We learned a lot about how group work should be done and that working on a project is far more time demanding and complicated than it sounds. Nevertheless we managed to keep our enthusiasm and interest in the topic throughout the whole course.

A moment that that inspired us all is when we got the chance to visit the center for visually impaired children in the town of Grave. It was an opportunity for which we are very grateful and it was all possible with the help of Don van Dijk, who was very enthusiastic about us making a project related to his work and passion - helping blind and visually impaired people. Going to the educational center gave us a first look in the lives of visually impaired children and how brave, optimistic and ambitious they are. Without a doubt that single meeting we had motivated us to not only work on the project because of the course but also because we believed our idea is a just one and that it can lead to future improvements and innovations that would benefit those people.

The process of coming up with the idea for our project and the realization of it was a long and with many obstacles along the way: from lack of desired hardware to the need of research and the desire we had to come up with something that is unique and innovative. We all learned new an important things during the course of the project, from teamwork, organization and time management to working with Android Studios and the information and knowledge needed to make an application for a user group that is so unique and foreign to the smartphone game sector. The feedback we got from our test cases tells is that people find our idea interesting and helpful. In our eyes it is not just a university project but also an aspect of education for visually impaired that can be further improved. It is something non of us will forget and we might work on it further if the situation and opportunity arises.

In the end we are all pleased with the journey we went on and the outcome of it. We are satisfied with the work we have done and the lessons we have learned, which will without a doubt be useful in our future.

References

  1. ↑ Willemsen, D.R.S. (2015). Designing Haptic Graphics for Mathematics: Towards Accessible Math Education for Blind Students. TU Delft. Retrieved from https://repository.tudelft.nl/islandora/object/uuid:b613feb9-7460-49c4-bd93-cc917d84f108?collection=education
  2. ↑ 2.0 2.1 Marti-Parreño, José & Galbis-Córdova, Amparo & Miquel, María. (2017). Students’ Attitude towards the Use of Educational Video Games to Develop Competencies. Computers in Human Behavior. 81. 10.1016/j.chb.2017.12.017.
  3. ↑ Perrotta, C., Featherstone, G., Aston, H. and Houghton, E. (2013). Game-based Learning: Latest Evidence and Future Directions (NFER Research Programme: Innovation in Education).
  4. ↑ Ekman, I., Ermi, L., Lahti, J., Nummela, J., Lankoski, P., & Mäyrä, F. (2005, June). Designing sound for a pervasive mobile game. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology (pp. 110-116). ACM.

Coaching Questions Group 2

Retrieved from "https://dsdwiki.wtb.tue.nl/index.php?title=Second_project:_Educative_game_for_blind_children&oldid=54464"