PRE2018 3 Group15

Group members

Name	Student ID
Sanne van den Aker	1258788
Leon Cavé	1240614
Robin van Tol	1246240
Heleen Fischer	1223688
Laura Barendsz	1245706

Introduction

In 2017 2.5% of Dutch parents thought their child between 6 and 12 has an autism spectrum disorder (ASD). This means around 40000 children in the Netherlands have difficulty interacting on a social level and maintaining structure. Since we know there is no conventional 'cure' for ASD, a lot of research has been done on finding the right therapy to help these children through life. As we now live in a technology based era, therapy has also shifted in this direction. In the last ten years, a lot of research has been done on the help of social robots during therapy. Especially NAO is a popular robot on this front. Even though there are promising results that show social robots can be very effective, there is still a long way to go before they can be fully implemented in therapy and maybe even daily life.

Problem Statement

Children with ASD often struggle with social interactions, something which is important to deal with as these individuals may have a high potential but cannot fully participate in society due to the lack in social skills. It is thus important these people get appropriate therapy to help them with this. In the last few years, it has been shown that this therapy is more effective when guided or mediated by a social robot. A possible explanation for this is that they are less intense and have a more procedural way of interacting. Because of this, interactions are always very similar and sameness is appreciated by these individuals. However, currently most of these findings come from tests with one on one therapy, but to really fully participate in society, it is important that people know how to collaborate, which is not really simulated in these cases. There are some cases were NAO is used in lego therapy, which shows the usefulness of group therapy and stimulation of collaboration. However here the children do not have the same role, which can cause agitation. Aside from this, later in daily life, these children will often have to work with colleagues that have an equal role and they will have to be able to collaborate with them. Thus, the next important thing to look into next is using a robot like NAO for group therapy where children are equal to each other as a means to improve social interaction and mainly collaboration.

Objectives

For this project a test group is needed, in our case, we will need children with autism in the age group from 6 to 12. To reach this group we will go to a middle school for children with autism. For our project we will use a Nao robot which will be lent to us. The robot’s task is to help the children play a game. The robot will give them a task where the children physically have to do what Nao has told them to do. Nao will give them an assignment where the children have to use their bodies to create a figure, this way the children need to collaborate to achieve the goal and have to physically touch.

To properly test the robot a rubric will be made which will be discussed with the test group. This will give an insight on the actual practicality of the robot and will help answer the question if it is a helpful aid for children with autism. Different tasks will be measured by grading the specific task.

To achieve this goal, we will use a Nao robot which is provided for us, the robot needs to be programmed to perform the tasks that are set out and the program needs to be altered according to the test results.

There are a few solutions already available for children with autism, however we would like to achieve something new in that it is a form of group therapy using a robot. The robot should help a child on multiple levels while remaining approachable by its appearance.

The project must be finished in 8 weeks, in order to help us meet this deadline we made a planning.

Users

Main users

For our main users we confined to children between 6 and 12 years of age which is around the age for primary school in the Netherlands who have autism spectrum disorder (ASD). Their needs will be defined as the needs they have of improving their social interactivity skills by means of education to help cope with their neurological disorder. Children with ASD are more vulnerable to emotional and behavioural difficulties and can become anxious in social situations:

“Children with developmental disorders such as specific language impairment (SLI) and autism spectrum disorders (ASD) appear to be more vulnerable to emotional and behavioural difficulties than typically developing children.” Wenche Andersen Hellanda, Turid Helland (2017) [1]

Another article [18] states 3 reasons why children with ASD become anxious:

1. Continuous social rejection leading to an increase in the anxiety levels in people with ASD.

2. Awareness of social deficiencies.

3. Lack of flexibility in social situations and confusion of social stimuli.

This suggests great care and caution is needed when handling these children which is something to be considered in the education of children with ASD, however we will (for now) mostly confine ourselves to the educational needs of children with ASD to help them increase their social skills by means of a robot. Many studies have already been done on the use of robots for educational purposes and many proposed methods proved an effective tool to improve the social skills of these children (see literature study). Some of these studies even stated the robots to be more effective at attracting attention while teaching [2]. Thus these robots seem to be in line with the needs of the main users when implemented correctly.

Less direct needs of the child can also be described such as the need for entertainment which will improve the attention of the children thus helping his/her motivation and thus education. These have already been addressed by some articles for example using cartoon like robots [3] or music in therapy [4].

Secondary users

Our secondary users are people who are in some way connected to the product but not directly such as parents, teachers and other school personnel. The needs of the parents can be described as the need for education of their children; if a parent registers their child for the social robot therapy he/she expects the robot to improve the child’s social skills. The parents also have the need of trust for the social robot in a safety aspect; will the robot therapy be as safe as normal therapy at all times? These questions are hard to directly answer but solutions may be easily implemented such as adding communication to the school director’s office or local emergency services in case of emergency.

Other school’s personnel need can also be described as the need for trust in the robot, safety and everyone who works directly with the robots has needs related to ergonomics. We will not focus on these aspects for now mostly because the robot we will use has been provided.

Autism

Autism or autism spectrum disorder (ASD) is an chronic disability. People suffering from this are challenged by social skills, repetitive behaviors, speech and nonverbal communication. There are many different subtypes of autism, most influenced by a combination of genetic and environmental factors. Each person has therefore his own strength and challenges. The ways in which people live can range from completely independent to people who require significant support in their daily live. This depends on their learning capabilities. Some people are highly skilled while others are severely challenged.

The cause of autism is still unknown, but a combination of genetic and environmental factors can increase the risk. If autism is a frequent disease in a family the children are more likely to be born with autism. Changes in certain genes increases the risk that a child will develop autism. These genes can be passed on by the parents or arise spontaneously in an early embryo, sperm or egg. The gene changes by themselves do not cause autism they just increase the risk. Some environmental factors can increase or decrease the risk of autism. When the parents age is advanced the risk to get a child with autism get increased because they genes will mutate more easily. Also pregnancy and birth complications such as extreme prematurity and low birth weight increase the risk of autism. Last the time between pregnancies, if it is less than one year apart, can increase the changes of an autistic child. To decrease the risk of autism prenatal vitamins containing folic acid can be taken by the mother before, at conception and through the pregnancy. [5]

Autism is characterized by deficient social interaction, poor communication skills and abnormal play patterns. One of the earliest visible characteristics of this illness is avoiding eye contact. They do this because looking others in the eye is uncomfortable or even stressful for them. They also fail to see the emotional state of others and do not understand that their actions might affect the feelings of others. Nearly everyone with autism has some level of language impairment. This can range from complete lack of verbal communication to people who talk incessantly and do not allow others to add to the conversation. People with autism struggle with self-initiated interactions. Instead of asking for food when they are hungry they resort to a tantrum. Another thing that can cause a tantrum with an autistic child is changing the planning. Children like to have a structure live and therefore want to know everything beforehand.

Early diagnosis and intervention show a great long-term effect. People who are in therapy from a young age are more capable to deal with problems later. Therapists are generally unable to diagnose autism by children younger than 3. [6]

During our project we will focus on improving the social skills. Autistic children like to learn in physical ways. This will be used to learn them how to collaborate. Collaborating is a voluntary relationship that requires shared responsibility and works to achieve a common goal. For autistic children this could be a real struggle. When it is improved they can participate in more recreational activities, this is associated with a increased quality of life. The ability to collaborate is essential in order to integrate into society. This will make them more valuable to society in later life.[7]

Group therapy

In recent years a great interest has began to grow in applying robots in group therapy aimed at children with ASD(Autism Spectrum Disorder). Socially interactive robots are used to communicate, perceive emotions and interpret reactions. Social Robots are being used to teach children with ASD skills and to play with them. Social robots are appealing to children with ASD because of the predictability of a robot compared to humans[8]. Robots always act according to their programming, which means that a robot will act the same way in a situation. Humans can be affected by their emotions and might act differently one day than any other day. A robot will also not use complex facial expressions which can be hard to interpret for children with ASD. This can make communicating with a robot easier than communicating with other humans. For a child with ASD, it is important that when a task is executed correctly the child is rewarded by a comment, the child will then be encouraged to complete the same task more often. This way a routine is made for the child which can be very helpful for them. Group therapy can teach a child with ASD to have social skills[9], this is a very important thing, because when children learn to be social earlier in life they can more easily fit in when they’re older. A form of group therapy can be through art because children with ASD are often very visually thinkers. Studies have shown that group therapy can decrease anxiety in children with ASD[10]. Different types of group therapy for children with ASD already exist, when focusing on a specific form, LEGO therapy with a humanoid robot[11]. This form of game-based therapy has positive effects on training social skills. A few problems were found with this therapy, the children find it hard to keep focused for a long time. Also in this form of therapy the children had unequal roles, which lead to some frustration and jealousy. In this project the roles will be equal between the children and only Nao will have a different role, the mediator. This will be one of the criteria for this project. The game will take a short time with a lot of new assignment to keep the children interested.

NAO

What is NAO?

NAO is a programmable and humanoid robot which has a height of 58 centimeters. The first NAO was created by SoftBank in 2006. Since then NAO is evolved and the 6th version of NAO was launched in 2018. During this time NAO became a standard in education and research. Furthermore it is used as an assistant by companies to welcome, inform and entertain visitors. The latest version of NAO has 25 degrees of freedom. This ensures that he can adapt to the environment and move easily. To locate himself in space in a stable manner NAO uses an inertial measurement unit. This a electronic device that consist of an accelerometer, a gyroscope and four ultrasonic sensor. These sensors are placed on his hand, feet and head. NAO also has 4 directional microphones and speakers to interact with humans. Speech recognition and dialogue in 20 language are also implemented in the robot. Another important feature of the robot are his 2D cameras to recognize shapes, objects and people. But one of the most important things is that NAO is open and fully programmable.

Academic and scientific use

NAO is a very broad used application. Over 200 academic institutions worldwide have made use of the robot. By the end of 2014 5000 robots were used within educational and research institutions in 70 countries. The use of NAO differ a lot among the institutions. One of the first times NAO was used was for the RoboCup. Later the robot became a well working application and was used for more advanced research. Nowadays the robot is often used to do research into human-robot interactions. A French institute tested a system or robotic autobiographical memory with NAO. NAO should eventually train international space station crews and assist elderly patients. Currently a platform started to enhance the use of NAO in elementary school to learn the children program early on their live.

Software

NAO comes with embedded software and desktop software. Due to the two systems the robot has autonomous behavior but the robot can also be remotely controlled. OpenNAO is the operating system of the robot. This program is specially developed for the needs of NAO. It is an embedded GNU/Linux distribution based on Gentoo. OpenNAO controls NAOqi which is the main software that runs on the robot. With this software the robot can be controlled by calling modules. The software can also be used on a computer with a simulated robot. Choregraphe is the desktop software. This is a visual programming language. Behaviors can be created without writing a code. An extra Choregraphe behavior can be programmed with the use of Python. This behaviors can be created and tested on a simulated robot before using them with the real NAO. This programming also allows to monitor and control NAO.

ALVisionRecognition

The game we want to play needs image recognition to make it a success. Choregraphe already has a function which can do this, namely ALVisionRecognition. The robot tries to recognize different pictures, objects sides and locations learned previously. There are some limitations to this. NAO recognizes the object of it’s key points, so if the object does not have good texture it will become difficult. Another important limitation is that the robot is not able to recognize classes but object instances. The last limitation is that currently the key points recognized are only matched with one of the learned key points. This means that the choice between two objects can be difficult if they are too similar. A good recognition is dependent of the light condition, rotation, distance and angles. The distance may not be less then the half of the distance used for learning. The distance may also not be more than twice the learning distance. The angle can be up to 50° inclination compared with the angle the robot learned the object from. The video sensor of the is used in combination with the video monitor panel to recognize the objects. The video monitor shows the figure and the programmer has to draw the contour of the object. This way a database of objects can be created. When the programmer is a bit experienced the robot can learn new objects in less than 30 seconds.

Appearance

Children prefer a robot with less human like facial expressions, this is because the appeal of a robot is that it is less complex than a human. Humans tend to use facial expressions as a form of communication which can be hard for children with ASD to interpret. However studies have also shown that a robot with doll-like features and human like face appeals to children with ASD and that they make eye contact with the robot [12]. When taking the torso of the robot into account, children appeared more interested in the robot when it appeared more human like. This was simulated by dressing the robot up in clothing to hide the robots mechanical features. The appearance did not affect the outcome of the test but did keep the children interested which is a valuable factor[13]. As per intonation of the robot’s voice the conclusion is less clear. When a prerecorded voice is used, the robot will sound more human which has been proven to appeal to children[14]. However this method is very time consuming and difficult, since the robot cannot change the dialogue. Since the intonation of the robot do not seem to have an effect on the children’s task performance, it does not seem worth the time to work with an prerecorded voice. Since the true effects of a humanized robot versus a mechanic robot in appearance is in it’s test results negligible

Learning goals

During our project we want children to imitate a certain figure, mainly a letter. The National University of Singapore already conducted a research where children learned the alphabet through full body interactions. 4-7 years old played word out, an interactive game. During this game they learned the alphabet through play but also learned to collaborate.

People with ASD have problems with imitation. There are different types of imitation, like actions with objects, gestures and body movement, and sound or words. During our project we are focusing on the imitation of body movement. This is very important to improve because children will have problems with language outcomes, play skills, joint attention and peer play.

Another experiment, with a collaborative puzzle, is used to improve collaboration with autistic children. This puzzle is displayed on an computer screen because innovative technologies seem to be useful for people with ASD. This research pointed out that children with autism can focus better on a computer screen where only the necessary information is displayed. This way they don’t get distracted by unnecessary stimuli. Computers are also free of social demands and this also helps autistic people with focusing on their task. Because of the advantages of the computer the children could mainly focus on the collaborating part. The result of this research was therefore positive. Children enjoined the game and the collaborating part didn’t bother them.

Combining these three experiments we think our experiment could be a success. We use a robot so the children don’t have to focus on the different gestures of people. They play a game and will imitate figures together to improve their collaboration skills as well as imitation skills. Which will conclude in children who are better in collaborating but also have better play skills and could therefore better play with other children.

Approach

The goal of our project is to help young autistic children with their social behavior. We want to do this with the help of a robot NAO.

To determine the needs of the children a literature study will be performed. This study also includes research about what is already done, so the state-of-the-art is explored.

With the information obtained the code for the robot will be written. This will be done in either in TiViPe or Choregraphe depending on the complexity of our project.

NAO will be used to teach the autistic children some social skills. The children will be in a group of 3-4 people. We choose for this option to increase collaboration skills. Research done before provide the evidence that group therapy for autistic child’s work. One on one therapy with NAO gives also desirable results. We want to combine those things to improve the skills of the children.

NAO is going to give a instruction to the children to imitate a certain figure. The children have to recreate this figure working together. The difficult level can be increased according to the age or impairment of the autism. The robot could start with showing the figure on a tablet. At the end the robot could give them an assignment to solve. An example can be: ‘create the answer of the question 3 + 4’.

If the children are not able to solve the problem NAO should give them a hint and motivate them. NAO should also determine if the problems are too simple or too difficult. This could be determined by the time they need to deal with it.

When the code of the robot is finished we will test it with autistic children and teachers of an elementary school. We are limited with the testing because of the short time of this course. Therefor we can only test once and we will mainly focus on the opinions of the experienced teachers of the school. The results and feedback will be implemented after this. At the end of the project, the robot will be demonstrated with all the adjustments mentioned by the children and teachers.

Scenarios

In order to get an overview of certain situations that might develop 5 different scenarios were thought of:

Scenario 1

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS (Childhood Autism Rating Scale) [1] and an SQR score of above 15 thus having ASD [2]. NAO is going to play a game with these children where the children have to make simple shapes that NAO verbally provides. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand, all the children nod as if they understand and the game can begin. The setup for the game is in a room with a soft pink floor making it easier for NAO to recognize the shapes the children make. A carpet of a constant bright color was also brought for if the color scheme of the floor would not allow easy shape recognition or if the floor would be uncomfortable to lie on. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘H’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. NAO inspects the shape while standing atop a table (around 72 cm in height) for a better overview, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together” and after the children lie down NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then NAO uses its internal clock to decide that it is time to stop the exercise. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positively.

Scenario 2

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS (Childhood Autism Rating Scale) [1] and an SQR score of above 15 thus having ASD [2]. One of the children has a CARS score between 38 and 60 showing a heavy form of autism [1]. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘K’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. The more heavily autistic child gets anxious and does stops collaboration with the other children. The spectating school personnel intervenes and the child is excused. Thus the game will continue with one child less. NAO’s software is quickly adapted by the technical staff and the possible shapes NAO could choose have been altered to insure the shapes are possible to imitate with one child less. The children are informed and continue the game by forming the first shape NAO tells them, the letter ‘P’. NAO inspects the shape while standing atop a table which is rather small thus its shape recognition is poorly accurate, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together”. NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then the technical staff stops the experiment because the altering of NAO’s program upset its internal clock. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positive but states that NAO might require a different setting for more sensitive children.

Scenario 3

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS (Childhood Autism Rating Scale) [1] and an SQR score of above 15 thus having ASD [2]. Two of the children has a CARS score between 38 and 60 thus both showing a heavy form of autism [1]. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if someone does not understand they should raise their hand, some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘R’ using their bodies, the children stand around and don’t begin making the shape, so Nao asks them: ”Would you all sit down and then decide between yourselves who will be which part of the shape”. Two of the kids sit down but the two with heavy forms of ASD refuse to participate and show signs of anxiety. The spectating school personnel intervene and both children are excused. The remaining two children are divided into groups that have not yet participated and Nao’s program is reset in preparation for the next group.

Milestones

• Narrow down problem.
• State of the art research completed.
• Scenarios and test plan completed.
• Learn to use Nao.
• Code completed and tested.
• Interview completed.
• Final demonstration completed.
• Wiki page completed.

Planning

• Week 2:

• Introduction; Sanne
• Problem statement; Sanne & Laura
• User’s and RPC's; Leon
• Milestones; Robin
• Add planning and headings to the wiki; Robin
• State-of-the-art literature study; Everyone
• Write smart objectives; Heleen
• Approach; Laura

• Week 3:

• Literature study; Heleen & Laura
• Write scenarios; Robin & Leon
• Interview Emilia; Everyone
• Find people for the interview; Sanne

• Week 4:

• Literature study; Heleen & Laura
• Start programming; Robin & Leon
• Test plan; Sanne

• Week 5:

• Make a working version; Robin & Leon
• Test plan; Sanne, Heleen & Laura
• Update wiki; Heleen & Laura

• Week 6:

• User test + interview; Sanne, Leon & Robin
• Adjust code; Leon & Robin
• Process results; Heleen, Laura & Sanne

• Week 7:

• Process code feedback; Leon & Robin
• Write a conclusion and update wiki; Heleen & Laura
• Make presentation; Sanne

• Week 8:

• Provide presentation; Leon, Sanne & Heleen

Literature review

Currently, these are the summaries of the state of the art articles

1. Group therapy can be used to relax defences and increase participation. Structuring a group psychotherapy sessions can be problematic. Those children often seem to be ego-centric individuals where the give and take needed for a productive group dynamic can be difficult. Some therapist want to achieve a good dynamic in the group. This can become confusing because a serious content becomes masked by fun-oriented formats. Giving idoms as stimulus serves as an open-ended structure within both the session’s verbal and artmaking components. Most outcomes had the effect of camouflaging criticism by infusing an element of levity and playfulness to the work that comes with the group therapy process.

2. Art therapy and cognitive-behavioral techniques in a group therapy format are used to help children with social developmental disorders to improve their social skills. Scores revealed a significant improvement in assertion scores, coupled with decreased internalizing behaviors, hyperactivity scores and problem behavior scores in student.

3. The result of this research shows that the basic HRI (human robot interaction) carried out by NAO is able to suppress the child’s autistic behavior during the child-robot interaction. Also, more eye contact is observed between the child and robot compared to the child with his teacher during regular class sessions. The IQ of the children participating in this research falls in the moderate catergory.

4. An robot can function as a friendly playmate, as a behavior eliciting agent, as a social mediator, as a social actor, as a personal therapist. The main advantages are that robots are less complex than humans. They are more predictable and it is easier for children to follow instructions from a robot than a human. The children will not be overwhelmed by the complexity of verbal and non-verbal communication. Robots also make embodied interactions possible. Due to their physical affordances, interactions involving tactile exploration and physical movements make the robots more engaging and interesting for children. Robots are less intimidating than humans. Robots can be used as small, colorful toys, ensuring that children feel at ease during the interaction.

5. Autistic children have problems with several things like self-initiated interactions, turn-taking activities, imitation, emotion recognition, joint attention and triadic interactions.

6. Parents and therapists are enthusiastic to work with robots such as NAO as it encourages children to participate on a social level. Especially for children with ASD as NAO is not impulsive and gives a certain structure. The main problem at this time is related to voice recognition.

7. Game-based therapy have positive effects on training social skills, especially with lego. Children in the experiment did not like a slow speaking robot. The repetition of the rules also irritated them. Therapists prefer one on one therapy with a robot and a child, instead of a pair of children as a robot cannot adapt well to situations happening between the two children. One clear outcome is that children (with ASD?) react more strongly to the verbal than non-verbal communication, thus the speech should be improved most. Major challenge is keeping children with ASD focused, especially when the task at hand takes long. However, robots are considered exciting by the children, which makes it easier to keep their focus.

8. For this study children with ASD were paired with their siblings. The idea was that a robot mediated lego therapy, which is supposed to improve collaborative behaviors. However no significant changes in behavior was found (as opposed to lego therapy without a robot mediator). Even though robot therapy is advised for children with ASD, it comes with limitations. One of the main limitations is that the robot has a limited behavioral repertoire, so no differentiation in prompt levels could be made. The article thus recommends the increasing of the variability in prompt levels to respond to children’s individual needs.

9. The goal of this study was to see whether children with ASD performed better when NAO is more humanlike and when his intonation differs. It looked also for a difference in likeability. There was no evidence that intonation has significant effect on performance. However, as the children in the study where already familiar with NAO and it’s voice Jasmijn, the children may not have experienced possible positive influences of a different intonation as individuals with ASD like things to be the same (thus preferring Jasmijn). Bodily appearance does also not influence the performance. It does however has a large effect on the affective state of an individual. It also helps when the bodily appearance matches the intonation. So mechanical + monotonous or human-like + normal. For these congruent combinations, the happiness also increases.

10. The article mentions that there are cases where individuals with ASD react better to robots and prefer interaction with robots over humans. However it also says that studies that come to these findings have very inconsistent results. A lot of studies are too exploratory or theoretical. It concludes that the robots have great potential in a clinical setting, with advantages like the appeal of technology to children with ASD, the ability of a robot to have a simple social behavior (that is easier to understand and focus on) and a robot can be easily adapted for each individual. However further research should be done to determine the validity. Work should be evaluated by experts with clinical expertise as most studies are now focused on the technology development and not necessarily clinical application.

11. Bibi Huskens, Rianne Verschuur, Jan Gillesen, Robert Didden & Emilia Barakova (2013) Promoting question-asking in school-aged children with autism spectrum disorders: Effectiveness of a robot intervention compared to a human-trainer intervention, Developmental Neurorehabilitation, 16:5, 345-356, DOI: 10.3109/17518423.2012.739212

This research aimed at investigating applied behavior analysis (ABA)-based interventions either done by a human trainer or by a robot to promote self-initiated question asking in children with autism spectrum disorder (ASD). In an ABA analysis it is studied how behaviour works in real life situations, with the goal to increase behaviour considered “good” and decrease behaviour that are harmful or affect learning. This research focused mainly on two research questions: (a) are an intervention conducted by a robot and an intervention conducted by a human trainer effective in promoting question-asking in children with ASD? and (b) which of these two interventions is more effective in promoting question-asking in children with ASD? The research in this article was done with the help of six children participants which were between 8-14 years old, had a ASD diagnosis, an IQ of above 80 on a standardized test and where not able to initiate a question after a statement of a trainer. A NAO robot was used and each child had two interventions by both a robot and a human trainer while being under video surveillance. The results of this article state that both robot and human-trainer ABA-based intervention is an effective tool that increased self-initiated question asking in children with ASD between baseline and the first intervention and was maintained during the follow up. The article states that no conclusions could be drawn with regard to the differential effectiveness of the robot or human-trainer interventions.

12. Felippe Sartoratoa, Leon Przybylowskia, Diana K. Sarko (2016) Improving therapeutic outcomes in autism spectrum disorders: Enhancing social communication and sensory processing through the use of interactive robots.

This research focuses at examining a range of socially interactive robots which are currently the most used for therapeutic purposes and their therapeutic effects. The researchers discussed how enhanced sensory processing and integration of social cues into these robots may underlie the benefits that these robots bring about. They state these robots and their interactions might provide therapeutic benefits by allowing the audiovisual cues in social interactions to be experienced in a simplified version of human interaction. The research focusses mainly on two parts: The deficits in autism spectrum disorder (ASD), consisting of a sensory perception and neurobiology part, and the spectrum of socially-assistive robots, where multiple types of robots are examined. They conclude that ASD comes with perception impairments related to multisensory integration of more complex stimuli. Furthermore they state that ASD comes with several neurobiological overarching impairments such as one involving the mirror neuron system which facilitates imitation and social communication, or deficits in the amygdala which might underlie social cognition deficits. After examining the impairments that come with ASD the research focusses on the spectrum of available socially-assistive robots, where they investigate 4 types of robots: humanoid robots, cartoonish robots, animal robots and robots in robotic form. They conclude that humanoid robots were found to elicit enhanced generalization of social skills taught during therapy, however non humanoid robots attract the most attentional engagement and robots with a simple appearance increase levels of interaction and are more easily accepted by children with ASD. Next the research examined the application of neuroscience tools to improve the therapeutic value of the robots and concluded that the underlying mechanism behind the benefits of these robot interactions remains largely unknown and state that recent studies in their lab have begun to address this question. The research does not end with a general conclusion but with a written part about the access to social robot therapies and state that there is a lack of affordable, commercially available robots for in home use and that there is also a lack of longitudinal studies addressing following children’s process as they go through childhood.

13. Sang-Seok Yun, Hyuksoo Kim, JongSuk Choi, Sung-Kee Park (2015) A robot-assisted behavioral intervention system for children with autism spectrum disorders.

This research aims at examining the feasibility of a robot-assisted intervention system for the training of children with autism spectrum disorder (ASD) via human robot interaction (HRI) based on the discrete trial teaching (DTT) protocol. Their proposed robot architecture configures 4 modules: human perception, user input, the interaction manager and the robot effect, with which the robot can generate different training stimuli and can automatically cope with the child’s responses by using human recognition and interaction techniques. The article consists of 3 main parts: Robot system for autism treatment, experimental setup and results and discussion. In the robot system for autism treatment part they discussed their interaction architecture and its treatment protocols which focusses mainly on the social interaction skill of eye contact and reading emotion. Furthermore they discussed their robot automatic interaction system which involved sound, physical movement made by the robot, displayable content and even material reward. The experimental setup part goes into great detail about the setup used in the research, stating they used two different robots for their tests and a monitoring setup. Their results verify the effectiveness of behavioral intervention in autism treatment using a robot and they claim to be confident that the proposed system they used can be attributed to the positive effect of social skill training in children with ASD.

14. Jaishankar Bharatharaj, Loulin Huang, Christian Krägeloh, Mohan Rajesh Elara, Ahmed Al-Jumaily (2018) Social engagement of children with autism spectrum disorder in interaction with a parrot-inspired therapeutic robot.

This paper discusses a series of pilot studies in which ten participants (children with ASD) were to engage with a parrot inspired therapeutic robot. Hereby it was evaluated if these children with autism spectrum disorder (ASD) exhibited more social interaction when engaging with the robot compared to another human. The article states that parrots already have a widespread use in therapeutic sessions for example patients with post-traumatic stress disorder, the researches give this as a reason to design a parrot inspired therapeutic robot. The participants were gathered in a study room and three sets of outcomes were collected; one at baseline, one without human and robot interaction and one with robot or human interaction. Data was collected on 12 defined behaviours such as: looking at the person, going close to the person or touching the person. The study’s results indicate a significant improvement in children’s interaction abilities as opposed to sessions with human involvement and the 12 types of social interaction abilities monitored reported that the robot has the potential to act as a social robot to improve social interactions in children with ASD. The article ends with the notion that the sample group was small and the study was not conducted long-term and there were no follow ups, hereby stating they aim to address these limitations by increasing the scale and duration of their research.

15. Wing-Chee So⁎, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition.

This study aims to compare the learning outcomes in children with autism spectrum disorder (ASD) and intellectual disabilities from robot based interventions on gestural use versus the human based interventions. A intervention protocol was designed, implemented and tested on a test group of 23 children aged six to 12 divided randomly in two groups; one for human-based intervention and one for robot-based interaction. Two NAO (Aldebaran Robotics Company) robots were used and programmed to speak and/or produce 14 gestures. The treatments consisted of pretests, four training sessions (twice per week), an immediate posttest and a follow up posttests after two weeks. The learning outcomes in gestural production and recognition did not differ between the robot- and human-based intervention groups. The research however suggested that the children in the robot-based intervention group were more likely to establish eye contact with the teachers than those in the human-based intervention groups. To conclude the researchers state that there does not seem to be a difference between the likeliness of children with ASD who received human-based gestural training to recognize gestures and children who received robot-based gestural training.

19. This article presents an overview of robots that are currently used in therapy and education for children with ASD and to indicate what objectives they address.

20. This article shows research done about the attention span of children when interacting with a robot compared to therapists.

21. In this work a preliminary evaluation of an innovative social robot-based treatment for subjects with ASD is described.

22. This article reviews studies in robots and autism as a neurodevelopmental disorder that impacts social communication development, and the ways social robots could help children with autism develop social skills.

23. This article is about how group-based behavioral therapy may help children and teens with high-functioning autism spectrum disorder (ASD) develop stronger and longer-lasting social skills, according to a new trial of group classes for children with ASD developed at Goethe University Frankfurt in Germany.

State of the Art sources

• [1] David R. Henley PhD, ATR (2000) Blessings in Disguise: Idiomatic Expression as a Stimulus in Group Art Therapy with Children, Art Therapy, 17:4, 270-275, DOI: 10.1080/07421656.2000.10129762

• [2] Brook, Marie. (2008). Outcome-Based Evaluation of a Social Skills Program Using Art Therapy and Group Therapy for Children on the Autism Spectrum. Children & Schools. 30. 10.1093/cs/30.1.27.

• [3] S. Shamsuddin, H. Yussof, L.I. Ismail, S.Mohamed, F.A. Hanapiah, N.I. ZahariInitial (2012) response in HRI-a case study on evaluation of child with autism spectrum disorder interacting with a humanoid robot Nao. Procedia Engineering, 41 (2012), pp. 1448-1455

• [4] Cabibihan, JJ., Javed, H., Ang, M. et al. Int J of Soc Robotics (2013) 5: 593. https://doi.org/10.1007/s12369-013-0202-2

• [5] J Ricks, Daniel & Colton, Mark. (2010). Trends and Considerations in Robot-Assisted Autism Therapy. Proceedings - IEEE International Conference on Robotics and Automation. 4354 - 4359. 10.1109/ROBOT.2010.5509327.

• [6] A. Amanatiadis, V. G. Kaburlasos, C. Dardani, S. Chatzichristofis (2017), Interactive social robots in special education, Proc. IEEE 7th Intl. Conf. on Consumer Electronics (ICCE), 210-213.

• [7] Barakova, E. I., Bajracharya, P., Willemsen, M. H. J., Lourens, T., & Huskens, B. (2015). Long-term LEGO therapy with humanoid robot for children with ASD. Expert Systems, 32(6), 698-709

• [8] Huskens, B., Palmen, A., Van der Werff, M., Lourens, T., & Barakova, E. (2015). Improving collaborative play between children with autism spectrum disorders and their siblings : the effectiveness of a robot-mediated intervention based on lego (R) therapy. Journal of Autism and Developmental Disorders, 45(11), 3746-3755.

• [9] van Straten, C. L. C., Smeekens, I., Barakova, E. I., Glennon, J., Buitelaar, J. K., & Chen, A. (2018). Effects of robots’ intonation and bodily appearance on robot-mediated communicative treatment outcomes for children with autism spectrum disorder. Personal and Ubiquitous Computing, 22(2), 379-390.

• [10] Diehl, J. J., Schmitt, L. M., Villano, M., & Crowell, C. R. (2012). The Clinical Use of Robots for Individuals with Autism Spectrum Disorders: A Critical Review. Research in autism spectrum disorders, 6(1), 249-262.

• [11] Bibi Huskens, Rianne Verschuur, Jan Gillesen, Robert Didden & Emilia Barakova (2013) Promoting question-asking in school-aged children with autism spectrum disorders: Effectiveness of a robot intervention compared to a human-trainer intervention, Developmental Neurorehabilitation, 16:5, 345-356

• [12] F. Sartorato, L. Przybylowski, D. K. Sarko (2017) Improving therapeutic outcomes in autism spectrum disorders: enhancing social communication and sensory processing through the use of interactive robots, Journal of Psychiatric Research,90, 1-11

• [13] Yun SS, Kim H, Choi J, Park SK (2016) A robot-assisted behavioral intervention system for children with autism spectrum disorders, Robotics and Autonomous Systems 76:58-67.

• [14] Jaishankar Bharatharaj, Loulin Huang, Christian Krägeloh, Mohan Rajesh Elara, Ahmed Al-Jumaily (2018) Social engagement of children with autism spectrum disorder in interaction with a parrot-inspired therapeutic robot, Procedia Computer Science, 133, 368-376.

• [15] Wing-Chee So, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition, Research in Developmental Disabilities, 86, 62-75.

• [16] Wenche Andersen Hellanda, Turid Helland (2017) Emotional and behavioural needs in children with specific language impairment and in children with autism spectrum disorder: The importance of pragmatic language impairment

• [17] Samata R.Sharma, XeniaGonda, Frank I.Tarazi (October 2018) Autism Spectrum Disorder: Classification, diagnosis and therapy

• [18] Christine K. Syriopoulou Delli, Stavroula A.Polychronopoulou, Gerasimos A.Kolaitis, Alexandros-Stamatios G.Antoniou (December 2018) Review of interventions for the management of anxiety symptoms in children with ASD

• [19] Huijnen CAGJ, Lexis MAS, de Witte LP (2017) Robots as New Tools in Therapy and Education for Children with Autism. Int J Neurorehabilitation 4:278. doi:10.4172/2376-0281.1000278

• [20] Waltz, E. (2018, August 09). Therapy Robot Teaches Social Skills to Children With Autism. Retrieved from https://spectrum.ieee.org/the-human-os/biomedical/devices/robot-therapy-for-autism

• [21] D. Mazzei et al., "Robotic Social Therapy on Children with Autism: Preliminary Evaluation through Multi-parametric Analysis," 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, Amsterdam, 2012, pp. 955-960.doi: 10.1109/SocialCom-PASSAT.2012.99

• [22] Richardson, Kathleen & Coeckelbergh, Mark & Wakunuma, Kutoma & Billing, Erik & Ziemke, Tom & Gomez Esteban, Pablo & Vanderborght, Bram & Belpaeme, Tony. (2018). Robot Enhanced Therapy for Children with Autism (DREAM): A Social Model of Autism. IEEE Technology and Society Magazine. 37. 30-39. 10.1109/MTS.2018.2795096

• [23] Pedersen, T. (2018, August 08). Group Therapy Can Aid Social Skills in Kids with High-Functioning Autism. Retrieved from https://psychcentral.com/news/2016/01/26/group-therapy-can-aid-social-skills-in-kids-with-high-functioning-autism/98194.html

• [24] Shakernaeeni H, Chowdhury U.The Structure of Group Therapy for Anxiety in Children and Adolescents with Autism Spectrum Disorder in a CAMHS Setting in the UK. J Child Dev Disord. 2018, 4:1

• [25] Mirella Dapretto, Mari S Davies, Jennifer H Pfeifer, Ashley A Scott, Marian Sigman, Susan Y Bookheimer and Marco Iacoboni. Understanding emotions in others: mirror neuron dysfunction in children with autism spectrum disorders; doi: 10.1038/nn1611

• [26] Kristie Brown Lofland. The Use of Technology in Treatment of Autism Spectrum Disorders

Test plan

Testing beforehand Ideally, NAO can autonomously play the game with the children without our interference. If this cannot be done, NAO will be controlled, but this does not has the preference as this might influence the child’s behavior when they notice the robot does not actually act on its own. To see whether it is possible to let NAO autonomously play the game, we have to see how complex the implementation of NAO can be, so first several tests should be done during the programming. These tests include seeing how well it reacts to voices and if it can recognize whether the children are working together or not talking with each other at all. For example, it would be nice to have encouragement of NAO when the children do not dare to talk with each other. Another part which has to be looked into is at which height NAO can best check the figure made, which is important even if NAO is being controlled. This height should not be higher than the average child however, so NAO does not accidentally come across as intimidating. So for this, the very first test is to determine a minimum height for NAO to scan the floor well enough. This test will be as follows: NAO is placed on the ground and it is checked whether it can determine a shape on the ground roughly the size of the figures the children will make. If this is not successful, NAO will be placed higher, in steps of 10 cm until it is. The height at which NAO first is successful will be the height used in the following tests.

After this the tests for the autonomous using of NAO will be done, the first test will be: There is as less as possible background noise. NAO asks how many children will play with him. Three is answered. Next is checked whether NAO gives a figure that is preprogrammed to be made by three children. The test will be repeated to come to a correct result.

If this test fails, it does not make sense to follow up with a test that checks whether children are working together, as this will give different and more unclear sounds which will be harder to pick up and interpret. If the first test is a success, the next stage consists out of multiple tests. These tests will show whether NAO gives the right reaction to different scenarios.

Test a) The background noise should be at a minimum. NAO will give a figure. First, two or more people should discuss what they are going to do and perform this action correctly. NAO should not encourage or give hints as everything is going well. At the end NAO should give a compliment and eventually go on to the next exercise.

Test b) The background noise should be at a minimum. NAO will give a figure. Two or more people should discuss their plan and perform the action, however the action should not be done correctly. NAO should not encourage, as there is collaboration, but it should give the group a hint on what to do, as the figure is not correct. After this hint, the group should correct the figure. Now, NAO should give a compliment and go on to the next exercise.

Test c) The background noise should be at a minimum. NAO will give a figure. Two or more people do not say or do anything. As NAO can see them and hear they are not discussing, it should encourage the people to come up with a solution and perform this. After this the figure will be discussed and performed and NAO will check this. Then he gives a compliments and goes on to the next exercise.

If all these tests work out, NAO should be able to perform the game autonomously. If not, he will be controlled during the interaction with the children.

Actual testing with children This is at this stage a draft of what would be the ideal case, but is subject to change as we learn more about NAO. First off we want to have a small interview with the children and their supervisor to determine some small things like problems they might have with the test and to discuss with the therapist the scale of ASD.

For this a questionnaire still has to be set up.

After the questions are asked, NAO will be presented to the children and the test will be explained to them by NAO. NAO will be presented with clothing and a natural voice as, even though monotonous speaking may be more effective, NAO with clothing also has a positive effect on children and in that case it is important to match voice with appearance as this is shown to work better than a clothed NAO with a mechanical voice. First off NAO will ask how many children are participating, to make sure figures asked can be made with the amount of children present. After this NAO will tell the children which figure to imitate. Where necessary NAO will encourage the children and give hints. When coming to the right solution, it will compliment them and give a new figure. This will be repeated until the children have made three figures. At this point NAO will thank the children for playing with him and showing collaborative behavior. After this test, the children will be asked according to a questionnaire what their thoughts are. The supervisor will also be asked about his thoughts, as he will know more about the normal behavior of the children.

The questionnaire still has to be set up.

The test will be repeated with several groups.

References

• [1] Wenche Andersen Hellanda, Turid Helland (2017) Emotional and behavioural needs in children with specific language impairment and in children with autism spectrum disorder: The importance of pragmatic language impairment

• [2] Wing-Chee So, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition, Research in Developmental Disabilities, 86, 62-75.

• [3] F. Sartorato, L. Przybylowski, D. K. Sarko (2017) Improving therapeutic outcomes in autism spectrum disorders: enhancing social communication and sensory processing through the use of interactive robots, Journal of Psychiatric Research,90, 1-11

• [4] Samata R.Sharma, XeniaGonda, Frank I.Tarazi (October 2018) Autism Spectrum Disorder: Classification, diagnosis and therapy

• [5] J Ricks, Daniel & Colton, Mark. (2010). Trends and Considerations in Robot-Assisted Autism Therapy. Proceedings - IEEE International Conference on Robotics and Automation. 4354 - 4359. 10.1109/ROBOT.2010.5509327.

• [6] Autism speaks (2019). What causes autism? https://www.autismspeaks.org/what-causes-autism

• [7] Potvin, Marie-Christine MHS; Prelock, Patricia A.; Snider, Laurie; Collaborating to support meaningful participation in recreational activities of children with autism spectrum disorder. Topics in Language Disorders. Supporting Social Communication, Perspective Taking, and Participation in Children With Autism Spectrum Disorders. 28(4):365-374, October/December 2008.

• [8]: John-John Cabibihan, Hifza Javed, Marcelo Ang Jr and Sharifah Mariam Aljunied, “Why Robots? A Survey on the Roles and Benefits of Social Robots for the Therapy of Children with Autism” International Journal of Social Robotics, 2013, 5(4), 593-618, doi 10.1007/s12369-013-0202-2
• [9] Traci Pedersen (2018). Group Therapy Can Aid Social Skills in Kids with High-Functioning Autism https://psychcentral.com/news/2016/01/26/group-therapy-can-aid-social-skills-in-kids-with-high-functioning-autism/98194.html
• [10]Judoth A. Reaven, Audrey Blakeley-Smith, Shana Nichols, Meena Dasari, Erin Flanigan, Susan Hepburn; Cognitive-Behavioral Group Treatment for Anxiety Symptoms in Children With High-Functioning Autism Spectrum Disorders
• [11]Emilia I. Barakova, Prina Bajracharya, Marije Willemsen, Tino Lourens, Bibi Huskens. Long‐term LEGO therapy with humanoid robot for children with ASD.
• [12]Aude Billard PhD, Ben Robins PhD, Jacqueline Nadel PhD & Kerstin Dautenhahn PhD (2007) Building Robota, a Mini-Humanoid Robot for the Rehabilitation of Children With Autism, Assistive Technology, 19:1, 37-49, DOI: 10.1080/10400435.2007.10131864
• [13]van Straten, C.L., Smeekens, I., Barakova, E. et al. Pers Ubiquit Comput (2018) 22: 379.
• [14]Keith C Radley, Evan H. Dart, Kate A. Helbig, Stefanie R. Schrieber, Mary E. Ware. (2018) An evaluation of the additive effects of lag schedules of reinforcement. Developmental Neurorehabilitation 0:0, pages 1-12.