PRE2017 3 Groep4

Museum Tour Robot

Vision

The museum robot is an artificially intelligent robot guide to enhance a new visitors’ experience in a museum by providing an interactive experience. By conducting a dialogue, the museum visitor is shown around the museum. The robot adjusts his dialogue and guided tour to the visitor. This will be done on the basis of age, gender, interests and cultural background. The robot gets this information through a small introduction dialogue. On the base of this dialogue, every person gets a unique tour through the museum. During the tour, the robot is able to respond to questions asked by the visitors and he must also be able to act on certain actions. In addition to answering questions, the robot will also take in account the background, preferences and interest of the visitors. He can respond on it by giving recommendations about what the visitors must have seen on the basis of their interests. This way of touring will serve to attract more younger visitors to museums. This because the tour is more personalized and the robot will take in account the preferences of the visitors. Younger people are grown up with smartphones and technology. Therefore this way of touring will lead to more younger visitors. The robot is also able to give a tour in another language.

Since, we are aware that we only have 7 weeks to accomplish the above, we will only focus on the dialogue and the interaction between one visitor of the age between 18-30 years and Pepper. We will assume that the robot is perfectly able to guide the visitor through the museum, that the robot is aware of its position and the position of the visitor relative to the art piece and that it knows what the visitor is looking at when asking a question, so the robot knows what museum piece the question addresses. We will also assume that the dialogue will be able to be held in another language and that the robot can address multiple people at the same time. Also, we will assume that the robot can read the emotion of the visitor, as well as the visitor's body language. As to help personalize the tour, enhancing the personalization of tour that we will implement in the conversation.

We plan to be able to take into account the background, preferences and interest of the visitors, as well as the robot being able to adjust its dialogue to the user, easy storytelling for children and more elaborate storytelling with more background knowledge for a somewhat older category of people.

Orientation robot

Objectives

When developing and researching the museum tour robot, the objectives give a clear direction of the goals. The objectives are the following;

• The robot must be able to enhance the museum experience by giving personal tour based on interest and background.
• The robot must be able to attract younger visitors to the museum by taking into account the preferences of the visitors.
• The robot must be able to conduct a personal dialogue with a user.
• The robot must be easy to use also by elderly people

Users

A robot in a social environment must take all other agents (users) into account. The users in this scenario are;

• Museum visitors (younger people 18-30 years old)
• Museum owners
• Museum employees
• Maintenance technicians
• Government

User Requirements

Visitors:

• Personal, positive experience
• Experience on their level / their background (age, home-country, language)

Museum owner(s):

• More visitors going to museum, people visiting museum more than once
• System is cheap to maintain
• To attract another target audience to the museum
• Enthusiastic visitors

Museum employees:

• The system does not interfere with the employees’ duties (does not get in the way)
• The system can easily be controlled by an employee

Maintenance technicians:

• System does not require much maintenance
• Maintenance is easy to perform (parts are accessible)

Task Environment

Task environments are the “problems” to which rational agents are the “solutions”. In designing an agent, the first task is to specify the task environment as fully as possible.

PEAS: Performance measure, environment, actuators, sensors.

Performance measure: Safe, fast (enough), personal for visitors, legal (rules museum), //(something with reacting well to visitors)

Environment: Entrance, visitors, museum staff, museum art, paths, other robots

Actuators: Moving, talking

Sensors: Location mechanism, cameras (see visitors/artworks), speech recognition system

fully observable vs partially observable ; agent can only observe visitors in the current room.

single agent vs multiagent ; There are more entities in the environment; visitors, other robots.

Deterministic vs stochastic ; movement of visitors is not to be exactly predicted.

Episodic vs sequential ; discussable. Episodic, because every room has new artworks and new information, which does not depend on previous room. But sequential, because visitors keep knowledge of previous room and might recall on that, or make links.

Static vs dynamic ; Environment changes while robot is thinking. People will keep moving and discuss/talk with each other (potentially about the artwork)

Discrete vs continuous

Known vs unknown ; this is about if the robot knows its environment, which is true in this case, since the robot has pre-knowledge about the artworks and the museum layout.

Persona’s

Emma is 20 years old. She is a student and does not have a lot of money to spend. Now that she can’t visit a museum for free anymore, she only wants to visit one if she thinks it is worth her money. As Emma has visited some museums when she was little, she knows what a museum typically looks like and she thinks the majority of museums is a bit boring, quiet and dusty. Emma is very interested in robotics and when she does visit a museum she likes to watch and learn about robotics and its history.

John is 26 years old. He works full-time for a company, 40 hours a week, but he is single at the moment and wants to fill his spare time with activities. As his friends are very busy with work and some already with their families, John has to fill some spare time by himself. When he was a kid, his parents took him to many museums. He thinks of going to museums as a good way to relax and as a good way to think. John is mainly interested in museums with a lot of artwork, it doesn’t really matter for him what art, but he dislikes reading. He also doesn't like a tour with a big group of people, so, he likes to go to museums by himself that don’t require him to read a lot of text.

Alice is 28 years old. She has a very flexible job in which she can decide when she wants to work. To take her mind of work she likes to visit a museum. She likes to be surprised and prefers to get a tour through the museum. This way she hears more about the background of the artwork and she feels this enhances her museum visit.

Scenario’s

When Emma is talking with her friends in a break, she hears one of her friends went to the Van Abbe museum and got a tour from a robot. She is very enthusiastic and convinces Emma and her other friends to go there once too. A couple of weeks later Emma has some spare time in the weekend and she thinks of her friend talking about the Van Abbe museum and the robot. Since Emma is a bit bored she decides to go there and experience it herself. The Van Abbe museum is not the typical museum she likes going to, but she is very interested in the robot. So, Emma goes to the Van Abbe and gets a tour from the robot. Emma enjoys her visit very much. She is amazed by the abilities of the robot. The robot looks very cute, and can have a sophisticated conversation with her about the artworks, while guiding her through the museum. With the personal tour she got, she learns that the Van Abbe has more to offer her than she initially thought as she actually enjoys modern art more than she thought she would. After the visit, Emma feels encouraged to go to a museum more often as she is convinced now that museums are not that boring actually, especially when there is a robot to guide her around, but she also learned she is interested in modern art.

As John has quite some spare time he has to fill himself, he decided to subscribe to the Museumkaart. Due to this subscription, he regularly gets an email with news regarding museums. This way, he gets informed about the Van Abbe museum and their new use of robots guiding people around the museum while conversing with the visitor. John has already visited the Van Abbe museum a few years ago, but he is very interested in the robot and in the new collections of the Van Abbe museum, so he decides to go. He is really excited about the robot, in particular in the robot’s abilities to converse with visitors. He hopes the robot does a good job in conversing as it would mean for him he wouldn’t have to get a tour with a big group to get more background information about the artworks or read it himself. During his visit, the robot meets his expectations, and John really enjoyed his visit. He learned a lot this time when compared to his previous visit, as the robot told him a lot about the artworks. Since he didn’t read much during his last visit, this really opened his eyes to some artworks. John really hopes many other museums will also start making use of this robot. He wouldn’t have to take a tour with a big group of people, neither does he have to read himself. With the robot, he can take his time to walk through the museum and can ask the robot about the background information when he wants it. So, he can enjoy his visit to the museum more without having to read but just by conversing with the robot.

During one of her work days, Alice feels a bit stuck. So, she decides to take a big break and go to the Van Abbe museum. When buying her ticket, she asks for a tour through the museum. The receptionist tells her that they have a new way of guiding people through the museum and that Alice can choose from a regular tour or a tour from a robot. As Alice has never heard of the tour with the robot but is curious to it, she decides to go for the tour with the robot. To Alice’s surprise, the robot does a really good job. The robot talks to her very naturally and tells her the exact things she would expect to hear on a regular tour. She is very surprised by the way the robot personalizes her tour. It shows her exactly what she likes to see and tells her what she wants to hear in terms of background information. After her visit, she asks the receptionist if there are more museums that make use of this robot to guide visitors around. He tells her they are the first with this robot, but that there are some museums that will follow soon and gives her the names of the museums. Alice feels great after her visit and feels inspired to continue with her job.

Orientation project

This section will be about the personal orientation of the project.

Approach

First, a literature review will be done to gather information about the current state-of-the-art in multiple disciplines. Of course it is important to have knowledge on similar ideas and how they were approached. The necessary technology to realise the technical implementation in the end has to be looked at. For that, it is important to look at the state-of-the-art of Artificial Intelligence, smart home systems, person localization systems, chatbots, virtual personal assistants (like Siri from Apple), speech language processing systems, and probably more. Further, literature about user experiences in museums is necessary to see what the users do and do not like and be able to respond on these findings. After this broad state-of-the-art a more specific state-of-the art will be done.

Secondly, a study will be performed on the general tours in museums. This study will be done in a museum. The results of this study will be used to program the robot. We will use the Pepper robot to program and test.

Finally, the technical implementation phase will start by creating an implementation plan. The technical implementation will consist of three parts: A video of the Pepper robot who has a conversation with a user Program code with documentation A small report on the wiki

If everything goes well according to the planning, an optional test phase will be added after realizing the prototype. In this test phase, a small experiment study will be performed. Participants will experience the robot dialogue. After experiencing, the participants will fill in a small questionnaire on their experience and opinion.

Planning

The planning can be found on this page: Planning_Group_4.

Milestones

• Have a subject and a plan (week 1)
• Contact Margot Neggers (week 2)
• Definition ‘useful conversation’ (week 2)
• Contact museum (week 3)
• Appointment with Margot Neggers (week 3)
• Appointment with museum (week 4)
• Start coding of Pepper (week 4)
• Pepper answering a question (week 5)
• Pepper being able to converse in a museum context (week 6)

Deliverables

• Have a movie of Pepper in action
• Have first tests with Pepper
• Have a final presentation

State of the art

In this section, the current situation and most recent ideas and methods will be discussed. All used articles and sources can be found at the end of the wiki. Researching this "state of the art" will help to discover which steps can be made, and can help focus on more specific aspects of the museum tour robot.

First, the focus will be on more broad aspects around the museum tour robots. Topics include Museum/AI experience, person localization systems, Speech recognition, natural language processing and tour guides in other locations than musea. All research can be found on the page Broad SotA.

After researching the broad SotA, its decided that the focus will be on having a useful dialogue between robot and human. This is why extra research is done about the state of the art of dialogue/interaction between robot and human. This research can be found on this page Specific SotA

Research

Hier mag de tekst geplaatst worden.

Conversation robot

In this section, there will be more focus on dialogue between robot and human, and some important topics will be discussed to specify the direction of the work that will be done.

What is a 'useful conversation'?

Before making decisions, and to make objectives and methods more clear, the definition "useful conversation" between robot and human, in the context of a museum, will be made clear.

A conversation is defined as oral communication between at least two person in which information is exchanged. In this project, we define a conversation as oral communication between a robot and a human in which information is exchanged.

In this project, a useful conversation between a robot and a user shall take place in the context of a museum. This means that the conversation will take place in the museum about information regarding the museum and its artwork. The conversation shall take into account the personal preferences of the user. If the robot does not have knowledge of something the user asks, it shall respond that it doesn’t have knowledge on that, but that it can share information on a different subject. Also, the robot shall react to a question in a way that the question is actually answered (with the exception of a question that is irrelevant in the context, in that case it will respond as described above). On top of that, the questions, answers and remarks, provided by the robot, have to be sensible (contribute the context, experience or information-basis) and the exchanged information has to be truthful.

Conclusion

A useful conversation is oral communication between a robot and a human in a museum. In this useful conversation, all questions, answers and remarks provided by the robot are:

• Relevant/useful in the context (museum)
• Truthful
• A question that suits an answer
• Fitting the background of the user (language, origins, age)

General conversation concepts

Museum related questions of the visitor that Pepper should be able to answer

Can you tell me where I can find the toilet?/ Where is the bathroom?

Where is the exit?

Where is the elevator?

What time does the museum close?/When does it close?

I am thirsty, where can I get a drink? / Can I get a drink somewhere? I am hungry, can I get some food here? / Is there a cafeteria/restaurant/cafe?

General conversation questions that Pepper could answer

What is your name?

My name is Pepper

How old are you? / What is your age?

Do you have brothers and sisters?

When the visitor asks nonrelevant questions, Pepper could respond:

Sorry, I don’t know much about topics that do not relate to this museum. But I can tell you more about these artworks!

General questions that Pepper could ask and responses that Pepper could give

What do you think of this painting/artwork?

Would you like to hear more?

Should I explain more?

Do you want to hear more?

Would you like to know what is special about this piece of art?

Do you have questions about this piece of art? I can probably answer them.

Any questions about this piece of art?

Oh that is a good question!

Well, let’s see …

That is a very difficult question, let me search my memory.

General comments that indicate the visitor wants to move to the next artwork

Let’s go / Let’s move / Can we see another artwork?/ Can we go to the next one?

Conversation 1: "The Night Watch"

Beginning of conversation, at the beginning of the tour

Welcome to the Rijksmuseum! My name is Pepper and I will be your guide through the museum.

Personalizing tour

Is there anything you would really like to see?

So on your right you can see ‘The Night Watch’ painted by Rembrandt van Rijn. Dutch most famous painter of the 17th century. The painting is actually called ‘Militia Company of District II under the Command of Captain Frans Bannick Cocq’. But, as you can see the painting is quite dark. This is, for one, because Rembrandt intended this, so he could emphasize certain parts with light. But, as the painting aged, it turned a bit darker too by discoloration of the varnish. This way, people thought the scene took place at night, and so it got the nickname ‘Night Watch’.

Questions that the user could ask in italics with the answer Pepper could give

• What is the style of the painting?

In this painting realism and symbolism are combined.

• Can you tell me more about the painting? / When was the painting painted?

The painting is painted between 1639 and 1642. At the time, Rembrandt was about 35 years old. The painting itself is 3 meters 63 by 4 meters 38. It has been displayed in the Rijksmuseum since 1885.

• What is special about this painting?

This is Rembrandt’s largest, most famous canvas. Rembrandt was the first to paint figures in a group portrait actually doing something, and this way, telling a story. Traditionally, this would have been a static military group portrait, but Rembrandt depicted the motion. And Rembrandts dramatic use of light and shadow is also special.

• So what can I see on the painting?

The painting was painted in 1642 for the Arquebusiers guild hall. This was one of several halls of Amsterdam’s civic guard, the city’s militia and police. The two man in the front are captain Frans Bannick Cocq and lieutenant Willem van Ruytenburgh. The captain, dressed in black, is telling his lieutenant, dressed in white, to start the company marching. The guardsmen are getting into formation. Rembrandt used the light to focus on particular details, like the captain’s gesturing hand and the young girl in the foreground.

Would you like to hear more?

Saskia, Rembrandts wife, died in the year Rembrandt was working on the Night Watch. She died just before the painting was finished. Story goes, that Rembrandt gave the young girl Saskia’s face in memory of his late wife. We know for sure that the girl symbolically stands for the members of the civic guard that gave order for this painting. That is why she wears a chicken on her belt with the claws clearly visible, representing the clauwenier, the arquebusiers, and just behind that, a little less visible, you can see a pistol representing clover, the sort of weapon the militia used. The claw of the chicken and the clover were the symbols of the militia: the Arquebusiers.

• Does Rembrandt have a signature? What’s Rembrandt’s signature?

Rembrandt does have a signature, he painted ‘Rembrandt f 1642, at the bottom step, to the right of the foot of the musketier in the centre, firing his weapon. The signature can be seen just below the feet of the young girl.

• Where did he paint it?

Rembrandt probably painted this painting in a gallery on the patio of his dwelling

Pepper

Tutorial

To work with pepper, a program needs to be made. This will be done in Choregraphe, which can be downloaded here . A tuturial about dialogue boxes and different possibilities of pepper's dialogue can be found here: Pepper_Tutorial.

Coaching Questions

Questions of the coach and answers of the group can be found on this page (Coaching Questions Group 4) every week.