PRE2018 1 Group3

0LAUK0 Group 3

Dirk Heldens 0969145

Geert van Wordragen 1024503

Islam Lakehal 1203310

Nosa Dielingen 1023912

Tristan de Waard 0996117

Subject

A store clerk robot

Deliverables

SotA research (25 papers)

Regular updates on the wiki

Surveys

Prototype

(Video)presentation

State of the art research

The use of robots in a retail environment like stores and supermarkets has been a frequent subject for research. This research can be subdivided into a variety of smaller fields:

Navigation of robots in indoor environments with people and/or other robots. (e.g. [1], [2], [3], [12], [17])

Navigation of the robot in a shopping environment is an essential function it should have. Several approaches to navigation were found: Articles here describe the use of models that describe buildings in storeys, rooms, a set of places in each room and connectors among these. There is made use of so-called ‘highways’ for pre-determined robot paths and ‘off-roads’ where the robot plans its own path [1]. A different approach is the use of a sensor space as elaborated on in [2]. Other articles describe a modeling approach that predicts the surrounding pedestrian’s actions so that the robot can develop its own path. One promising example is an agent-based modeling approach where surrounding pedestrians are assigned the behaviours interact, watch, curious, ignore, cautious and avoid [12].

Design of appropriate actuators. (e.g. [4], [5])

The robot should be able to move around various objects, these could be heavy or fragile. Article [4] describes the use of a force control parameter for robot grippers, so that fragile objects will not be damage by actuators. Article [5] emphasises the challenge of designing safe actuators for human-centred robotics. The articles states that by reducing the effective impedance while maintain high frequency torque capability in actuators, safety and performance requirements can be achieved.

Design of appropriate sensors. (e.g. [10], [15], [17])

Various sensors are needed, especially for localisation and navigation purposes but also for object recognition. According to [17] the advances in computer vision have led to an increase in the use of cameras as sensors. They are often combined with other sensors such as odometry or lasers. Omnidirectional sensors stand out in the richness of information they provide. These sensors, together with robust models of the environment are important for designing an autonomous mobile robot.

Object recognition (in a shop context). (e.g. [19], [21], [25])

(Camera) sensors could be used for object recognition, which is an essential task for this robot application. The robot should be able to distinguish a large variety of shopping goods and should be able to detect if the product is misaligned or missing in the shelves. Article [19] describes a vision system where the user can specify an object the robot has to find and bring. When the recognition result is shown, the user can provide additional information, such as point out mistakes. Article [21] proposes a novel method for obtaining product count directly from an image using a monocular camera. Article [25] describes a patrolling robot that detects misaligned and out of stock products and provides the store associates with alert messages.

The social or legislation issues that arise when robots enter the workspace. (e.g. [6], [7], [8], [11], [13])

Robots working alongside humans could pose safety issues as well as open up question on how robots should interact (verbally) with humans. Another problem is that the use of robots could make humans redundant in this field of the job market. Article [8] says that retail automation is essential in competitiveness, but could lead to the minimum-wage employees being redundant as the robots are far cheaper. Robot store clerks are likely to be a disruptive force for the retail industry, this article states. Article [11] emphasises more on self-aware robots that become a part of society (including the retail sector) where brands are used as self-expression. Article [13] describes a means for robots to detect human action to make the cooperation between humans and robots in the workspace more attractive.

Human-robot interactions during shopping activities. (e.g. [9], [14], [22])

A robot store clerk should also be able to interact with humans. Humans might want information about a product or want to know where it is located. Article [9] proposes a robotic shopping companion to help customers in their shopping activities. Furthermore, the robot collects the emotional state of people through social interactions and then use that to influence people’s buying decisions. Article [14] goes further with investigating ways in which robots can persuade people. This could be applied to the robot store clerk in persuading people to buy a certain product. Article [22] describes ways in which verbal output of a robot can be made more human-like by introducing context-aware conversational fillers.

A combination of above fields, applied to a designed robot. (e.g. [16], [18], [20], [23], [24])

These articles describe a fully working system of a robot working in a retail environment. Especially article [20] is a great example, where a system is built that automates data collection for surveying and monitoring the shelves. The robot here can monitor shelves autonomously or through tele-operation. It can automatically detect out of stock situations. According to this article it will improve customer satisfaction, as shelve products are filled more frequently. The deployment also would not require modifying the existing store infrastructure and has a short return-on-investment period.

SotA sources

Navigation

[1] Gert L. Andersen, Anders C. Christensen, Ole Ravn, Mobile Robot Navigation In Indoor Environments Using Highways And Off-roads, Institute for Automation. bldg. 326, Technical University of Denmark

[2] Narongdech Keeratipranon, Robot Navigation in Sensor Space, Faculty of Information Technology Queensland University of Technology

[3] Guizzo, E. Ackerman. E. (2015). iRobot Brings Visual Mapping and Navigation to the Roomba 980. IEEE Spectrum: Technology, Engineering, and Science News. Retrieved from https://spectrum.ieee.org/automaton/robotics/home-robots/irobot-brings-visual-mapping-and-navigation-to-the-roomba-980

[12] Usher, J. M., McCool, R., Strawderman, L., Carruth, D. W., Bethel, C. L., & May, D. C. (2017). Simulation modeling of pedestrian behavior in the presence of unmanned mobile robots. Simul. Modell. Pract. Theory, 75, 96–112. doi: 10.1016/j.simpat.2017.03.012

[17] Payá, L., Gil, A., & Reinoso, O. (2017). A state-of-the-art review on mapping and localization of mobile robots using omnidirectional vision sensors. Journal of Sensors, 2017. DOI: 10.1155/2017/3497650

Actuators

[4] Lauzier, N. (2018, September 04). Robot Gripper Force Control. Retrieved from https://blog.robotiq.com/bid/53319/Robot-Gripper-Force-Control

[5] Michael Zinn, Bernard Roth, Oussama Khatib J. Kenneth Salisbury. A New Actuation Mastrogiovanni, F., & Casalino, G. (2018). Flexible human-robot cooperation models for assisted shop-floor tasks.Approach for Human Friendly Robot Design. Department of Mechanical Engineering Stanford University

USE

[6] Heinzmann, J., & Zelinsky, A. (2000). Building Human-Friendly Robot Systems. SpringerLink, 305–312. doi: 10.1007/978-1-4471-0765-1_37

[7] Wisskirchen, G., Biacabe, B T. et al. Artificial Intelligence and Robotics and Their Impact on the Workplace, April 2017, IBA Global Employment Institute

[8] Romeo, J. How Will Robot Store Clerks Disrupt Retail? - Robotics Business Review. (2016, July 26). Retrieved from https://www.roboticsbusinessreview.com/supply-chain/how-will-robot-store-clerks-disrupt-retail

[11] Gonzalez-Jimenez, H. (2018). Taking the fiction out of science fiction: (Self-aware) robots and what they mean for society, retailers and marketers. Futures, 98, 49–56. doi: 10.1016/j.futures.2018.01.004

[13] Darvish, K., Wanderlingh, F., Bruno, B., Simetti, E., Mastrogiovanni, F., & Casalino, G. (2018). Flexible human–robot cooperation models for assisted shop-floor tasks. Mechatronics, 51, 97–114. doi: 10.1016/j.mechatronics.2018.03.006

Human Interaction

[9] Bertacchini, F., Bilotta, E., & Pantano, P. (2017). Shopping with a robotic companion. Computers in Human Behavior, 77, 382–395. doi: 10.1016/j.chb.2017.02.064

[14] Lee, S. A., & Liang, Y. (. (2018). Robotic foot-in-the-door: Using sequential-request persuasive strategies in human-robot interaction. Computers in Human Behavior. doi: 10.1016/j.chb.2018.08.026

[22] Gallé, M., Kynev, E., Monet, N., Gallé, M., Kynev, E., Monet, N., & Legras, C. (2017). Context-aware selection of multi-modal conversational fillers in human-robot dialogues. 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 317–322. doi: 10.1109/ROMAN.2017.8172320

Sensors

[10] Mantha, B. R. K., Menassa, C. C., & Kamat, V. R. (2018). Robotic data collection and simulation for evaluation of building retrofit performance. Autom. Constr., 92, 88–102. doi: 10.1016/j.autcon.2018.03.026

[15] Rajithkumar, B. K., Deepak, G. M., Uma, B. V., Hadimani, B. N., Darshan, A. R., & Kamble, C. R. (2018). Design and Development of Weight Sensors Based Smart Shopping Cart and Rack System for Shopping Malls. Mater. Today:. Proc., 5(4, Part 3), 10814–10820. doi: 10.1016/j.matpr.2017.12.367

Store clerk robot implementation (technical)

[16] Tomizawa, T., & Ohya, A. (2006, October). Remote shopping robot system,-development of a hand mechanism for grasping fresh foods in a supermarket. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on (pp. 4953-4958). IEEE. DOI: 10.1177/1729881417703569

[18] Cheng, C. H., Chen, C. Y., Liang, J. J., Tsai, T. N., Liu, C. Y., & Li, T. H. S. (2017, September). Design and implementation of prototype service robot for shopping in a supermarket. In Advanced Robotics and Intelligent Systems (ARIS), 2017 International Conference on (pp. 46-51). IEEE. DOI: 10.1109/ARIS.2017.8297181

[20] Kumar, S., Sharma, G., Kejriwal, N., Kumar, S., Sharma, G., Kejriwal, N., ...Chauhan, V. K. (2014). Remote retail monitoring and stock assessment using mobile robots. 2014 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2014.6869136

[23] Lin, T., Baron, M., Hallier, B., Lin, T., Baron, M., Hallier, B., ...Dugan, J. (2016). Design of a low-cost, open-source, humanoid robot companion for large retail spaces. 2016 IEEE Systems and Information Engineering Design Symposium (SIEDS), 66–71. doi: 10.1109/SIEDS.2016.7489329

[24] Kamei, K., Ikeda, T., Kidokoro, H., Kamei, K., Ikeda, T., Kidokoro, H., ...Hagita, N. (2011). Effectiveness of Cooperative Customer Navigation from Robots around a Retail Shop. 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, 235–241. doi: 10.1109/PASSAT/SocialCom.2011.173

Object recognition

[19] Makihara, Y., Takizawa, M., Shirai, Y., Makihara, Y., Takizawa, M., Shirai, Y., ...Shimada, N. (2002). Object recognition supported by user interaction for service robots. Object recognition supported by user interaction for service robots, 3, 561–564vol.3. doi: 10.1109/ICPR.2002.1048001

[21] Kejriwal, N., Garg, S., Kumar, S., Kejriwal, N., Garg, S., & Kumar, S. (2015). Product counting using images with application to robot-based retail stock assessment. 2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2015.7219676

[25] Agnihotram, G., Vepakomma, N., Trivedi, S., Agnihotram, G., Vepakomma, N., Trivedi, S., ...Kumar, R. (2017). Combination of Advanced Robotics and Computer Vision for Shelf Analytics in a Retail Store. 2017 International Conference on Information Technology (ICIT), 119–124. doi: 10.1109/ICIT.2017.13

Problem Statement

Food prices have been steadily growing since 2006 [BRON]. Because of this, the highly competitive supermarkets have had to increase prices tremendously. For example, butter and margarine prices have gone up with a staggering 54.9% over the past 10 years [BRON]. Over the past decade there has also been a vast increase in internet shopping, primarily for non-food products, but also online grocery shopping. For grocery shopping, only 2% of all revenue is from e-commerce but this number is growing. [BRON] According to the Food Marketing Institute (FMI) 49% of US consumers have done some form of online grocery shopping in the past three months. [BRON] This has led to brick-and-mortar retailers having a hard time to compete with internet retailers. A survey by Retail Dive [BRON] concludes that customers still prefer to physically go to a store, the primary reason being able to see, touch and feel products before buying. It is important to give supermarkets and other brick-and-mortar stores more chance to compete against e-commerce increasing customer satisfaction. Therefore, it can be beneficial for both customers and retailers to optimise brick-and-mortar stores.

A solution might be found by streamlining, increasing the efficiency and economizing the state-of-the-art supermarket and/or other brick-and-mortar stores by means of a robot. Since self check-out services already are cost efficient, enhance customer satisfaction and are already becoming more and more abundant in supermarkets [BRON], an emphasis will be laid on the role of a store clerk in the supermarket. This research proposes the design of a robot store-clerk that either assists the human store clerk in inconvenient tasks, or acts as a replacement for the store clerk. The options for navigating the robot in an efficient and safe manner in a shopping environment still remains a challenge, so it will be one of the major topics in this research. Apart from the expected positive economic effects for enterprises, the negative and positive effects on users and society will be investigated thoroughly as well. To investigate the user’s needs, two surveys will be conducted: one targeting the general consumer another targeting store clerks. To demonstrate a safe and efficient ground floor movement and several functions of a store clerk robot a prototype will be made. The consequences of employing a retail/store clerk robot for user, society and enterprise will be discussed in the following sections, looking at state-of-the art research and real-life applications/examples.

User aspects

User types

For the store clerk robot, several users can be identified: Supermarket store clerks

In the case that the robot works alongside human store clerks, they become the users as well. A certain cooperation between human store clerks and robots will be necessary.

Supermarket customers

Customers can ask questions about the location of certain products or ask if products are in stock / when they come back in stock.

The supermarket boss/manager

They ‘own’ the robot and while they do not use the robot, the replacement of staff members with robots does have an impact on retail management in general.

The general user requirements for a robot store clerk can be formulated now.

General user requirements

1. Safe and efficient ground floor robot movement

Both customers and store clerks do not want to be obstructed by the robot’s movement, so the robot should have mechanisms to avoid obstacles and humans. It should move efficiently to reach areas where tasks need to be completed as quickly as possible.

2. Efficient refilling of shop shelves

Out-of stock shelves will reduce shop turnover. Shop owners strive for a high turnover, so it is important that shelves are in stock.

3. Careful handling of shopping goods

Inflicting damage to shopping goods renders them unmarketable, which is detrimental for the shop’s turnover and thus undesired for shop owners.

4. Shop customers must be able to interact verbally with the robot

Shop customers need to be able to gather information about product location or stock.

5. Low cost maintenance of robot store clerk

The robot should require less maintenance cost than the average salary of a human store clerk.

6. Low error rate in detection of shopping goods

The robot should correctly detect shopping goods with a low error rate in order for it to do its tasks correctly, and increase customer satisfaction.

Now, the benefits of employing a robot store clerk will be discussed.

Store clerk benefits

Since it is most likely not possible in the near future that store clerks will be entirely replaced by robots, a hybrid situation in which store clerks work alongside robot companions is believable in the short-term future. So what are the benefits for store clerks when working with robot companions? The obvious benefit for store clerks then is that their job will require less tasks when a robot store clerk is introduced in supermarkets. The section about store clerk tasks and possible robot applications already describes several tasks that can be delegated relatively easy to a robot. Especially the first task of detecting where products are out of stock is easy to replace and real-world applications already exist. For example, having a robot like Tally would already take out the task of checking the shelves for refilling. This robot gives useful real-time statistics and alerts to store clerks so they can distribute their tasks more efficiently and gives them more time for customer interaction. Taking out ‘inconvenient’ or ‘boring’ store clerk tasks in general would probably make their job less mundane. The survey targeted to store clerks will give better insights into which tasks are ‘boring’ and how their job might be improved by means of a robot.

Customer benefits

What are supermarket customer benefits when employing a robot store-clerk?

Finding a product or location they are headed to, is something customers regularly need help with. This is not only the case in supermarkets but also in e.g. airports or hotels. The good thing is that this a very viable short-term application for robots, as this requires very little robot agility. It is evident that in order for this to be a benefit, user requirement #1 should be taken into account. Darvish, K et al. [13] have proposed a promising solution for this with a very flexible option for robot navigation with obstacle avoidance in a full reactive fashion. Since robots can do this task very efficiently it is thought that customers will benefit from this. Doering, N. et al. [BRON] did a study on customer satisfaction and shopping effectiveness when a robot helped in guiding users to a certain product. It concluded that seniors of 65 years or older were less effective using the product search interface than other sociodemographic groups. However, the overall customer satisfaction turned out high and the participants’ intention to use the robot in the future was high. Another benefit for customers is that robots can quickly access goods that are out of reach or stored behind glass. Customers no longer have to wait for a free member of supermarket staff, thus saving time. An example of this is seen in Best Buy’s Chloe robot. [BRON] Here, customers can choose the product they would like to buy and the robot arm grabs it for them. The survey targeted to customers will expand on possible benefits for customers and test whether above benefits are also perceived by the surveyed persons.

Supermarket manager benefits

Although a supermarket manager does not physically use the robot, the introduction of robot store clerks does have an impact on this person. Since less staff members will be present in the supermarket, a manager might have less work to do. For example, less information needs to be spread around to staff members, less staff meetings are necessary and less job interviews might be the result. According to Keeling, K et al. [BRON] the advances in retail robotics will lead to managers having to ‘think afresh in developing interactions relevant to the needs of the user.’

User disadvantages

The movement of a robot in the shopping aisles might be inconvenient for both customers and staff members alike, since it can get in the way for people with a shopping cart. Therefore, the most important aspect of a retail robot in general should be ground floor movement that disturbs customers and staff member as little as possible, while also maintaining a decent operating speed.

Society aspects

Loss of jobs

The most obvious consequence of any kind of automation is a loss of jobs. Since these robots will take over some part of the store clerk work, this will not be any different. Store clerkship is a popular part time job for teenagers, employing 11% of the Dutch part time working youth in 2015. [BRON]

Loss of social contact

Supermarkets are, in a way, a social place as well. Automating tasks such as helping customers find the products they need will reduce social contact [citation needed?] and loss of social contact can noticeably affect health. [BRON] The store clerks will also have less human colleagues to talk to.

Cheaper products

One of the main reasons why automation is being used is because it saves the owner money. While, as mentioned before, some of these savings are to the detriment of otherwise potential store clerks, they can also benefit society as a whole.

Enterprise aspects

The relevant enterprises are the supermarkets that the store clerk-helpers get introduced to. For these supermarkets their benefit from these robots should be an increase in profit. This can happen both from saving money by needing to hire less store clerks and by getting more customers because of better service provided by these robots.

Streamlining a supermarket:

Increasing efficiency: A robot will be able to perform some tasks a lot faster than humans so letting it do these will make things generally be done quicker, this can make the store more attractive to customers. Decreasing the amount of humans working there will also make a store easier to manage because a robot does not need to be told what to do and is always available.

Economizing: Less store clerks need to be paid for

When the store clerks are assisted by a robot they can work more efficiently so this would mean less store clerks have to be hired to do the same amount of work. The money saved on their wages is equal to their hourly wage. In the Netherlands this is equal to €2.50-€12.00 per hour per clerk. [bron] The money saved should compensate for the cost of maintaining the robot and the power used. The increase in profit should eventually make up the for the cost of purchasing the robot, how long this takes depends both on the cost of the robot and how much labor it saves.

begroting supermarkt, vervanging robots vs mensen nieuwe job mogelijkheden

Effects on competition: New technology is already providing different ways to shop for groceries. Home delivery has started to gain traction: the online grocery shopping market has taken until now to gain traction, currently constituting 6.9% of the UK’s grocery market, but projected to increase 68.3% to £17.6 bn by 2021. [bron]. In order To make investing in this robot profitable traditional supermarkets need to keep existing for some time, if supermarkets get phased out in favor of home delivery it would mean it might be better to not invest too much into their future. [bron] [bron] It is also worth considering that supermarkets may become smaller because of decrease in demand so the robot could be designed with this in mind. The extra convenience that a robot can provide could also help traditional supermarkets compete with the increasing competition.

Is this technology still viable in the long-term future when supermarkets might be replaced by magazines with home delivery?

Freeing up workspace with for example Best Buy’s Chloe robot

Using retail statistics to improve turnover with for example Tally robot

Objectives

First of all, the basic tasks of a store clerk will be described concisely. Then, it is investigated which tasks can be delegated to a robot looking at state-of-the-art robot technology. A literature study will be performed to examine the financial consequences for a supermarket and the impact on society and users when employing a robot store clerk. Legislation aspects and responsibility regarding a store clerk robot will be considered also. The outcomes of these studies provide several requirements, preferences and constraints for the robot design. The survey targeted to customers and store clerks will be used to define additional design requirements. Next, a robot store clerk design will be proposed based on the previously defined tasks and requirements. Subsequently, a prototype will be built to demonstrate the functioning of a robot store clerk in a test environment and to look at the overall viability of robot store clerks in practice. Finally, a conclusion will be drawn on the desirability of employing a robot store clerk in the long and short term future. The most important user requirements and their accommodation in the design will be listed.

Store clerk task description

The job description for a store clerk [BRON] gives the following tasks a store clerk has to do:

1. Detecting where products are missing in the shelves.

2. Restocking missing products in shelves with oldest product upfront (First-in-first-out system)

3. Making sure products and their labels are displayed correctly (Dutch: ‘spiegelen’)

4. Helping customers find products / answering their product related questions.

5. Helping out the cashier when products need to be replaced at the counter.

6. Unloading goods from the storage truck, before the supermarket opens.

7. Finding products in the storage and bringing them into the supermarket.

Possible store clerk robot tasks

Looking at state-of-the-art research papers, the store clerk tasks will now be assessed on their possibility of being carried out by a robot.

Task #1 can according to [20] be done by making use of a patrolling robot either tele-operated or autonomous that scans shelves for misaligned or out-of-stock products. In the case of detection, a real time alarm is sent out so further action can be taken by staff-members or, preferably, other robots. It should be noted that this system can also detect where and when tasks #2 and #3 need to be performed. The entire solution ensures reduced store clerk manpower needed for these task and costs around US$2000.

Task #2 will require a robot with advanced sensors and actuators that can take out a row of products and put the new product in the back. With the current state-of-the-art robot designs this is still a very inefficient task for a robot to do, so at the moment human store clerks will need to perform this task. However, if an entire row of products is labeled out-of-stock by the system described in the previous subsection a robot might be able to fill up the shelf by inserting a row of products. Berntorp. K et al. [BRON] have proposed a design for robot manipulators for use in retail stores but it is only able to move one object at a time. It is therefore thought that this task should still be carried out by human store clerks.

Task #3 There are already several robots on the market that detect misaligned products, for example Simbe Robotics’ Tally robot [BRON]. This robot gives staff members an overview of retail statistics, also alerting where products are misaligned. Staff members still have to do the task themselves, as there are no robots on the market that can perform this action.

Task #4 requires a way customers can interact with the robot. Cheng, C. H. et al. [18] have already proposed a solution for this by designing a graphic user interface where customers can choose from product pictures, so that the robot can guide them to the product’s location or provide product information. Moreover, the customer is also able to give the robot commands or ask questions through speech.

Task #5 requires the robot to detect the product that is given, and should then be told what replacement is needed. It should then be able to find both product’s locations and replace the products accordingly. This requires advanced sensors and actuators and this is probably not possible as of now, since current actuators are not capable enough.

Task #6 is possible through the design of a robot, however this design will differ a lot from a store clerk robot, so it should be seen as a different robot. This task might be accomplished with a forklift type robot that can do the basic task of unloading pallets from a storage truck. For low budget supermarkets where most products are presented in pallets rather than shelves this task is especially important, when now they can also perform the restocking task. The design of a pallet lifting robot is given by Garibotto, S. et al. [BRON] However, it should be noted that this design does not incorporate appropriate movement in environments with humans walking around.

Task #7 could be achieved through a similar approach as task #6 where pallets from the storage are brought into the supermarket so that store clerks can do their restocking tasks afterwards.