PRE2018 1 Group3

From Control Systems Technology Group
Jump to navigation Jump to search

0LAUK0 Group 3 members

Dirk Heldens 0969145

Geert van Wordragen 1024503

Islam Lakehal 1203310

Nosa Dielingen 1023912

Tristan de Waard 0996117


Deliverables

  • SotA research (25 papers)
  • Identifying a subject and problem statement
  • Thorough research on user requirements
  • Assessment of SotA for our application
  • Giving a solution to the problem
  • Regular updates on the wiki
  • Final presentation

State of the art research

The use of robots in a retail environment like stores and supermarkets has been a frequent subject for research. This research can be subdivided into a variety of smaller fields:

  • Navigation of robots in indoor environments with people and/or other robots. (e.g. [1], [2], [3], [12], [17])

Navigation of the robot in a shopping environment is an essential function it should have. Several approaches to navigation were found: Articles here describe the use of models that describe buildings in storeys, rooms, a set of places in each room and connectors among these. There is made use of so-called ‘highways’ for pre-determined robot paths and ‘off-roads’ where the robot plans its own path [1]. A different approach is the use of a sensor space as elaborated on in [2]. Other articles describe a modeling approach that predicts the surrounding pedestrian’s actions so that the robot can develop its own path. One promising example is an agent-based modeling approach where surrounding pedestrians are assigned the behaviours interact, watch, curious, ignore, cautious and avoid [12].

  • Design of appropriate actuators. (e.g. [4], [5])

The robot should be able to move around various objects, these could be heavy or fragile. Article [4] describes the use of a force control parameter for robot grippers, so that fragile objects will not be damage by actuators. Article [5] emphasises the challenge of designing safe actuators for human-centred robotics. The articles states that by reducing the effective impedance while maintain high frequency torque capability in actuators, safety and performance requirements can be achieved.

  • Design of appropriate sensors. (e.g. [10], [15], [17])

Various sensors are needed, especially for localisation and navigation purposes but also for object recognition. According to [17] the advances in computer vision have led to an increase in the use of cameras as sensors. They are often combined with other sensors such as odometry or lasers. Omnidirectional sensors stand out in the richness of information they provide. These sensors, together with robust models of the environment are important for designing an autonomous mobile robot.

  • Object recognition (in a shop context). (e.g. [19], [21], [25])

(Camera) sensors could be used for object recognition, which is an essential task for this robot application. The robot should be able to distinguish a large variety of shopping goods and should be able to detect if the product is misaligned or missing in the shelves. Article [19] describes a vision system where the user can specify an object the robot has to find and bring. When the recognition result is shown, the user can provide additional information, such as point out mistakes. Article [21] proposes a novel method for obtaining product count directly from an image using a monocular camera. Article [25] describes a patrolling robot that detects misaligned and out of stock products and provides the store associates with alert messages.

  • The social or legislation issues that arise when robots enter the workspace. (e.g. [6], [7], [8], [11], [13])

Robots working alongside humans could pose safety issues as well as open up question on how robots should interact (verbally) with humans. Another problem is that the use of robots could make humans redundant in this field of the job market. Article [8] says that retail automation is essential in competitiveness, but could lead to the minimum-wage employees being redundant as the robots are far cheaper. Robot store clerks are likely to be a disruptive force for the retail industry, this article states. Article [11] emphasises more on self-aware robots that become a part of society (including the retail sector) where brands are used as self-expression. Article [13] describes a means for robots to detect human action to make the cooperation between humans and robots in the workspace more attractive.

  • Human-robot interactions during shopping activities. (e.g. [9], [14], [22])

A robot store clerk should also be able to interact with humans. Humans might want information about a product or want to know where it is located. Article [9] proposes a robotic shopping companion to help customers in their shopping activities. Furthermore, the robot collects the emotional state of people through social interactions and then use that to influence people’s buying decisions. Article [14] goes further with investigating ways in which robots can persuade people. This could be applied to the robot store clerk in persuading people to buy a certain product. Article [22] describes ways in which verbal output of a robot can be made more human-like by introducing context-aware conversational fillers.

  • A combination of above fields, applied to a designed robot. (e.g. [16], [18], [20], [23], [24])

These articles describe a fully working system of a robot working in a retail environment. Especially article [20] is a great example, where a system is built that automates data collection for surveying and monitoring the shelves. The robot here can monitor shelves autonomously or through tele-operation. It can automatically detect out of stock situations. According to this article it will improve customer satisfaction, as shelve products are filled more frequently. The deployment also would not require modifying the existing store infrastructure and has a short return-on-investment period.


SotA sources

Navigation

[1] Gert L. Andersen, Anders C. Christensen, Ole Ravn, Mobile Robot Navigation In Indoor Environments Using Highways And Off-roads, Institute for Automation. bldg. 326, Technical University of Denmark

[2] Narongdech Keeratipranon, Robot Navigation in Sensor Space, Faculty of Information Technology Queensland University of Technology

[3] Guizzo, E. Ackerman. E. (2015). iRobot Brings Visual Mapping and Navigation to the Roomba 980. IEEE Spectrum: Technology, Engineering, and Science News. Retrieved from https://spectrum.ieee.org/automaton/robotics/home-robots/irobot-brings-visual-mapping-and-navigation-to-the-roomba-980

[12] Usher, J. M., McCool, R., Strawderman, L., Carruth, D. W., Bethel, C. L., & May, D. C. (2017). Simulation modeling of pedestrian behavior in the presence of unmanned mobile robots. Simul. Modell. Pract. Theory, 75, 96–112. doi: 10.1016/j.simpat.2017.03.012

[17] Payá, L., Gil, A., & Reinoso, O. (2017). A state-of-the-art review on mapping and localization of mobile robots using omnidirectional vision sensors. Journal of Sensors, 2017. DOI: 10.1155/2017/3497650


Actuators

[4] Lauzier, N. (2018, September 04). Robot Gripper Force Control. Retrieved from https://blog.robotiq.com/bid/53319/Robot-Gripper-Force-Control

[5] Michael Zinn, Bernard Roth, Oussama Khatib J. Kenneth Salisbury. A New Actuation Mastrogiovanni, F., & Casalino, G. (2018). Flexible human-robot cooperation models for assisted shop-floor tasks.Approach for Human Friendly Robot Design. Department of Mechanical Engineering Stanford University

USE

[6] Heinzmann, J., & Zelinsky, A. (2000). Building Human-Friendly Robot Systems. SpringerLink, 305–312. doi: 10.1007/978-1-4471-0765-1_37

[7] Wisskirchen, G., Biacabe, B T. et al. Artificial Intelligence and Robotics and Their Impact on the Workplace, April 2017, IBA Global Employment Institute

[8] Romeo, J. How Will Robot Store Clerks Disrupt Retail? - Robotics Business Review. (2016, July 26). Retrieved from https://www.roboticsbusinessreview.com/supply-chain/how-will-robot-store-clerks-disrupt-retail

[11] Gonzalez-Jimenez, H. (2018). Taking the fiction out of science fiction: (Self-aware) robots and what they mean for society, retailers and marketers. Futures, 98, 49–56. doi: 10.1016/j.futures.2018.01.004

[13] Darvish, K., Wanderlingh, F., Bruno, B., Simetti, E., Mastrogiovanni, F., & Casalino, G. (2018). Flexible human–robot cooperation models for assisted shop-floor tasks. Mechatronics, 51, 97–114. doi: 10.1016/j.mechatronics.2018.03.006

Human Interaction

[9] Bertacchini, F., Bilotta, E., & Pantano, P. (2017). Shopping with a robotic companion. Computers in Human Behavior, 77, 382–395. doi: 10.1016/j.chb.2017.02.064

[14] Lee, S. A., & Liang, Y. (. (2018). Robotic foot-in-the-door: Using sequential-request persuasive strategies in human-robot interaction. Computers in Human Behavior. doi: 10.1016/j.chb.2018.08.026

[22] Gallé, M., Kynev, E., Monet, N., Gallé, M., Kynev, E., Monet, N., & Legras, C. (2017). Context-aware selection of multi-modal conversational fillers in human-robot dialogues. 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 317–322. doi: 10.1109/ROMAN.2017.8172320

Sensors

[10] Mantha, B. R. K., Menassa, C. C., & Kamat, V. R. (2018). Robotic data collection and simulation for evaluation of building retrofit performance. Autom. Constr., 92, 88–102. doi: 10.1016/j.autcon.2018.03.026

[15] Rajithkumar, B. K., Deepak, G. M., Uma, B. V., Hadimani, B. N., Darshan, A. R., & Kamble, C. R. (2018). Design and Development of Weight Sensors Based Smart Shopping Cart and Rack System for Shopping Malls. Mater. Today:. Proc., 5(4, Part 3), 10814–10820. doi: 10.1016/j.matpr.2017.12.367

Store clerk robot implementation (technical)

[16] Tomizawa, T., & Ohya, A. (2006, October). Remote shopping robot system,-development of a hand mechanism for grasping fresh foods in a supermarket. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on (pp. 4953-4958). IEEE. DOI: 10.1177/1729881417703569

[18] Cheng, C. H., Chen, C. Y., Liang, J. J., Tsai, T. N., Liu, C. Y., & Li, T. H. S. (2017, September). Design and implementation of prototype service robot for shopping in a supermarket. In Advanced Robotics and Intelligent Systems (ARIS), 2017 International Conference on (pp. 46-51). IEEE. DOI: 10.1109/ARIS.2017.8297181

[20] Kumar, S., Sharma, G., Kejriwal, N., Kumar, S., Sharma, G., Kejriwal, N., ...Chauhan, V. K. (2014). Remote retail monitoring and stock assessment using mobile robots. 2014 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2014.6869136

[23] Lin, T., Baron, M., Hallier, B., Lin, T., Baron, M., Hallier, B., ...Dugan, J. (2016). Design of a low-cost, open-source, humanoid robot companion for large retail spaces. 2016 IEEE Systems and Information Engineering Design Symposium (SIEDS), 66–71. doi: 10.1109/SIEDS.2016.7489329

[24] Kamei, K., Ikeda, T., Kidokoro, H., Kamei, K., Ikeda, T., Kidokoro, H., ...Hagita, N. (2011). Effectiveness of Cooperative Customer Navigation from Robots around a Retail Shop. 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, 235–241. doi: 10.1109/PASSAT/SocialCom.2011.173

Object recognition

[19] Makihara, Y., Takizawa, M., Shirai, Y., Makihara, Y., Takizawa, M., Shirai, Y., ...Shimada, N. (2002). Object recognition supported by user interaction for service robots. Object recognition supported by user interaction for service robots, 3, 561–564vol.3. doi: 10.1109/ICPR.2002.1048001

[21] Kejriwal, N., Garg, S., Kumar, S., Kejriwal, N., Garg, S., & Kumar, S. (2015). Product counting using images with application to robot-based retail stock assessment. 2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2015.7219676

[25] Agnihotram, G., Vepakomma, N., Trivedi, S., Agnihotram, G., Vepakomma, N., Trivedi, S., ...Kumar, R. (2017). Combination of Advanced Robotics and Computer Vision for Shelf Analytics in a Retail Store. 2017 International Conference on Information Technology (ICIT), 119–124. doi: 10.1109/ICIT.2017.13

Subject

User-centred design of robot navigation for a robot store clerk application in supermarket environments


Introduction

Food prices have been steadily growing since 2006 [BRON]. Because of this, the highly competitive supermarkets have had to increase prices tremendously. For example, butter and margarine prices have gone up with a staggering 54.9% over the past 10 years [BRON]. Over the past decade there has also been a vast increase in internet shopping, primarily for non-food products, but also online grocery shopping. For grocery shopping, only 2% of all revenue is from e-commerce but this number is growing. [BRON] According to the Food Marketing Institute (FMI) 49% of US consumers have done some form of online grocery shopping in the past three months. [BRON] This has led to brick-and-mortar retailers having a hard time to compete with internet retailers. A survey by Retail Dive [BRON] concludes that customers still prefer to physically go to a store, the primary reason being able to see, touch and feel products before buying. It is important to give supermarkets and other brick-and-mortar stores more chance to compete against e-commerce increasing customer satisfaction. Therefore, it can be beneficial for both customers and retailers to optimise brick-and-mortar stores.


Problem Statement

For streamlining, increasing the efficiency and economizing the state-of-the-art supermarket and/or other brick-and-mortar stores, employing robots seems like a decent approach. Since self check-out services already are cost efficient, enhance customer satisfaction and are already becoming more and more abundant in supermarkets [BRON], the role of a store clerk is something that that can be optimised. Implementing a robot that either assists the human store clerk in inconvenient tasks, or acts as a replacement for the store clerk is a way to achieve this.

For implementing such robots the expected positive economic effects for enterprises and the negative and positive effects on users and society will be investigated. Designing a robot store clerk requires it to have several tasks. Some of these tasks on their own are already seen in state-of-the-art robots, however, a robot that can execute all these tasks combined is not present and at the moment difficult to design.

This research aims to provide a first step towards the design of a store clerk robot by tackling one out of the three most important design difficulties. The first problem encountered when designing a robot in a supermarket is navigation. The environment of a supermarket or shop has several difficulties and benefits for robot navigation which make it unique. The difficulties lie in the customers and staff members walking around, which the robot needs to avoid in a reactive manner that is desirable for the surrounding people. The benefits are the fact that supermarkets have a distinct layout of aisles that rarely changes and the equipment of (security) cameras. Presently, there are little robot navigation options taking into account these difficulties and making use of the benefits. This research aims to find a solution for robot navigation in a supermarket environment to be used in robot store clerks, taking into account the robot user requirements.

Objectives

First of all, the state-of-the-art for robot store clerks is looked at by providing a small, exploratory literature summary. Then, the basic tasks of a store clerk will be described concisely. Subsequently, it is investigated which tasks can be delegated to a robot, looking at state-of-the-art robot technology. A literature study will be performed to examine the financial consequences for a supermarket and the impact on society and users when employing a robot store clerk. As a result, the most important design difficulties for a robot store clerk will be presented. Robot navigation in supermarket environments will be elaborated on as follows:

  • A description of a supermarket task environment is given.
  • Important difficulties and benefits in a supermarket environment, which make it unique from e.g. a depot or other crowded environments will be investigated.
  • State-of-the-art mapping, object avoidance and navigation algorithms will be assessed on their viability of applying in supermarket environments.
  • A description is given on how to make use of the benefits of this environment in robot navigation.
  • Finding a way to make effective use of the already mounted (security) cameras in supermarkets or a way to enhance them to fulfill the needs of the objective.
  • Looking at how much is visible through these cameras and finding out how many cameras are necessary.
  • Finding out how to make use of image stitching.
  • Describing how much the supermarket environment needs to be adapted.
  • Finding/developing a navigation algorithm that is most efficient in environments with a basic static structure, like in a supermarket with aisles.
  • Looking at how this would change environment mapping.
  • Investigating the use of shop products (semantic properties of the environment) as localisation markers in a navigation algorithm.
  • Solutions for the difficulties in robot navigation that are inherently present in a supermarket environment will be given by looking at specific customer and staff member needs.
  • The best way in which objects and humans can be avoided in for them the most desirable way possible is investigated through user needs (looking at, for example, speed and distance).
  • The use of object recognition for making a distinction between objects and humans through geometric shapes, depth, heat etc.
  • Looking at extra robot navigation complexities when the tasks of store clerks are considered.
  • Investigating how a robot knows the location of and can navigate to a certain product, which it needs to restock.
  • Investigating in which way a store clerk robot should escort/follow/move alongside a human when it is asked where a certain product is.
  • The best complete solution for robot navigation in supermarket environments will be presented.
  • A discussion on possible (security) problems that come with this navigation solution.
  • Providing topics for further research.

User aspects

User types

For the store clerk robot, several users can be identified: Supermarket store clerks

In the case that the robot works alongside human store clerks, they become the users as well. A certain cooperation between human store clerks and robots will be necessary.

Supermarket customers

Customers can ask questions about the location of certain products or ask if products are in stock / when they come back in stock.

The supermarket boss/manager

They ‘own’ the robot and while they do not use the robot, the replacement of staff members with robots does have an impact on retail management in general.

The general user requirements for a robot store clerk can be formulated now.

General user requirements

1. Safe and efficient ground floor robot movement

Both customers and store clerks do not want to be obstructed by the robot’s movement, so the robot should have mechanisms to avoid obstacles and humans. It should move efficiently to reach areas where tasks need to be completed as quickly as possible.

2. Efficient refilling of shop shelves

Out-of stock shelves will reduce shop turnover. Shop owners strive for a high turnover, so it is important that shelves are in stock.

3. Careful handling of shopping goods

Inflicting damage to shopping goods renders them unmarketable, which is detrimental for the shop’s turnover and thus undesired for shop owners.

4. Shop customers must be able to interact verbally with the robot

Shop customers need to be able to gather information about product location or stock.

5. Low cost maintenance of robot store clerk

The robot should require less maintenance cost than the average salary of a human store clerk.

6. Low error rate in detection of shopping goods

The robot should correctly detect shopping goods with a low error rate in order for it to do its tasks correctly, and increase customer satisfaction.

Now, the benefits of employing a robot store clerk will be discussed.

Store clerk benefits

Since it is most likely not possible in the near future that store clerks will be entirely replaced by robots, a hybrid situation in which store clerks work alongside robot companions is believable in the short-term future. So what are the benefits for store clerks when working with robot companions? The obvious benefit for store clerks then is that their job will require less tasks when a robot store clerk is introduced in supermarkets. The section about store clerk tasks and possible robot applications already describes several tasks that can be delegated relatively easy to a robot. Especially the first task of detecting where products are out of stock is easy to replace and real-world applications already exist. For example, having a robot like Tally would already take out the task of checking the shelves for refilling. This robot gives useful real-time statistics and alerts to store clerks so they can distribute their tasks more efficiently and gives them more time for customer interaction. Taking out ‘inconvenient’ or ‘boring’ store clerk tasks in general would probably make their job less mundane. The survey targeted to store clerks will give better insights into which tasks are ‘boring’ and how their job might be improved by means of a robot.

Customer benefits

What are supermarket customer benefits when employing a robot store-clerk?

Finding a product or location they are headed to, is something customers regularly need help with. This is not only the case in supermarkets but also in e.g. airports or hotels. The good thing is that this a very viable short-term application for robots, as this requires very little robot agility. It is evident that in order for this to be a benefit, user requirement #1 should be taken into account. Darvish, K et al. [13] have proposed a promising solution for this with a very flexible option for robot navigation with obstacle avoidance in a full reactive fashion. Since robots can do this task very efficiently it is thought that customers will benefit from this. Doering, N. et al. [BRON] did a study on customer satisfaction and shopping effectiveness when a robot helped in guiding users to a certain product. It concluded that seniors of 65 years or older were less effective using the product search interface than other sociodemographic groups. However, the overall customer satisfaction turned out high and the participants’ intention to use the robot in the future was high. Another benefit for customers is that robots can quickly access goods that are out of reach or stored behind glass. Customers no longer have to wait for a free member of supermarket staff, thus saving time. An example of this is seen in Best Buy’s Chloe robot. [BRON] Here, customers can choose the product they would like to buy and the robot arm grabs it for them. The survey targeted to customers will expand on possible benefits for customers and test whether above benefits are also perceived by the surveyed persons.

Supermarket manager benefits

Although a supermarket manager does not physically use the robot, the introduction of robot store clerks does have an impact on this person. Since less staff members will be present in the supermarket, a manager might have less work to do. For example, less information needs to be spread around to staff members, less staff meetings are necessary and less job interviews might be the result. According to Keeling, K et al. [BRON] the advances in retail robotics will lead to managers having to ‘think afresh in developing interactions relevant to the needs of the user.’

User disadvantages

The movement of a robot in the shopping aisles might be inconvenient for both customers and staff members alike, since it can get in the way for people with a shopping cart. Therefore, the most important aspect of a retail robot in general should be ground floor movement that disturbs customers and staff member as little as possible, while also maintaining a decent operating speed.

Society aspects

Loss of jobs

The most obvious consequence of any sort of automation is a loss of jobs. Since these robots will take over some part of the store clerk work, this will not be any different: there will be less human store clerks required per establishment. This does, however, not necessarily have to lead to there also being significantly less jobs in total. For one thing, the robots will bring more work for manufacturing and maintenance. These are Store clerkship is a popular part time job for teenagers, employing 11% of the Dutch part time working youth in 2015. [BRON]

Loss of social contact

Supermarkets are, in a way, a social place as well. Automating tasks such as helping customers find the products they need will reduce social contact [citation needed?] and loss of social contact can noticeably affect health. [BRON] This means that until robots can fully fulfill this need for social contact (which they eventually will [bron]), we should be careful with exactly what tasks they replace. This still leaves another problem: if there are less human store clerks in an establishment, they will also have less human colleagues to talk to. This is more difficult to counteract, but the fact that a bigger part of a store clerk’s work will now consist of doing the social tasks robots which the robots are not yet ready for will certainly help. Something else to keep in mind is that smaller forms of automation such as self-scanning are already reducing social contact in supermarkets, so perhaps

Cheaper products

One of the main reasons why automation is being used is because it saves the owner money. While, as mentioned before, some of these savings are to the detriment of otherwise potential store clerks, they can also benefit society as a whole.

Enterprise aspects

The relevant enterprises are the supermarkets that the store clerk-helpers get introduced to. For these supermarkets their benefit from these robots should be an increase in profit. This can happen both from saving money by needing to hire less store clerks and by getting more customers because of better service provided by these robots.

Several benefits can be found by streamlining a supermarket using a store clerk robot.


Increase in efficiency

A robot will be able to perform some tasks a lot faster than humans so letting it do these will make things generally be done quicker, this can make the store more attractive to customers. Decreasing the amount of humans working there will also make a store easier to manage because a robot does not need to be told what to do and is always available.

Reduction of labour costs

Robot store clerks might replace certain staff members, so less store clerks need to be paid for.

When the store clerks are assisted by a robot they can work more efficiently so this would mean less store clerks have to be hired to do the same amount of work. The money saved on their wages is equal to their hourly wage. In the Netherlands this is equal to €2.50-€12.00 per hour per clerk. [bron] The money saved should compensate for the cost of maintaining the robot and the power used. The increase in profit should eventually make up the for the cost of purchasing the robot, how long this takes depends both on the cost of the robot and how much labor it saves.

begroting supermarkt, vervanging robots vs mensen nieuwe job mogelijkheden


Long-term viability

Is a robot store clerk still viable in the long-term future when supermarkets might be replaced by magazines with home delivery?

New technology is already providing different ways to shop for groceries. Home delivery has started to gain traction: “the online grocery shopping market has taken until now to gain traction, currently constituting 6.9% of the UK’s grocery market, but projected to increase 68.3% to £17.6 bn by 2021.” [bron]. In order To make investing in this robot profitable traditional supermarkets need to keep existing for some time, if supermarkets get phased out in favor of home delivery it would mean it might be better to not invest too much into their future. [bron] [bron] It is also worth considering that supermarkets may become smaller because of decrease in demand so the robot could be designed with this in mind. The extra convenience that a robot can provide could also help traditional supermarkets compete with the increasing competition.

Effects on supermarket competition

Under the customer benefits section, Best Buy’s Chloe robot was already introduced. This robot acts as a storage and retrieval system of over 15,000 products. A robot arm can grab products behind glass very quickly for customers. This also means that shelves can be larger vertically since customers do not need to reach for it. According to PaR systems [BRON], this solution has freed up 92.9 m2 of store floor space. More products can be stalled in the same store floor space. It also means an employee’s time is freed up to assist customers in other areas. Additionally, this solution looks at buying trends, so it is always stocked with the newest and relevant items according to individual store trends. Simbe Robotics’ Tally robot [BRON] is another example of a robot that collects retail statistics for a store. These functions can all be beneficial in competition.

Another example can be seen at Walmart, which is investing in inventory scanning robots developed by Bossa Nova. This robot looks at the depth of shelves and determines if a shelve is fully stocked, completely out of stock or some level in between. An alert is sent to store clerks to physically restock the shelves. This was done so labour costs can be decreased, gather more customers and halt lost revenue opportunities, all to better compete with internet retailers like Amazon. According to a CNBC study [BRON] overstocks and out-of-stocks cost retailers 1.1 trillion USD globally in lost revenue and the majority of the root causes of missed sales and bad planning was due to outdated technology and personnel laziness. CNBC claimed that ‘75% of US adults have come across an unavailable product in stores over the past year’ resulting in frustrated shoppers deciding to shop at another retailer or buy nothing. [BRON] All in all, retail robots definitely seem like a promising solution for brick-and-mortar stores’ competition with internet retailers.

Store clerk task description

The job description for a store clerk [BRON] gives the following tasks a store clerk has to do:

1. Detecting where products are missing in the shelves.

2. Restocking missing products in shelves with oldest product upfront (First-in-first-out system)

3. Making sure products and their labels are displayed correctly (Dutch: ‘spiegelen’)

4. Helping customers find products / answering their product related questions.

5. Helping out the cashier when products need to be replaced at the counter.

6. Unloading goods from the storage truck, before the supermarket opens.

7. Finding products in the storage and bringing them into the supermarket.

Possible store clerk robot tasks

Looking at state-of-the-art research papers, the store clerk tasks will now be assessed on their possibility of being carried out by a robot.

Task #1 can according to [20] be done by making use of a patrolling robot either tele-operated or autonomous that scans shelves for misaligned or out-of-stock products. In the case of detection, a real time alarm is sent out so further action can be taken by staff-members or, preferably, other robots. It should be noted that this system can also detect where and when tasks #2 and #3 need to be performed. The entire solution ensures reduced store clerk manpower needed for these task and costs around US$2000.

Task #2 will require a robot with advanced sensors and actuators that can take out a row of products and put the new product in the back. With the current state-of-the-art robot designs this is still a very inefficient task for a robot to do, so at the moment human store clerks will need to perform this task. However, if an entire row of products is labeled out-of-stock by the system described in the previous subsection a robot might be able to fill up the shelf by inserting a row of products. Berntorp. K et al. [BRON] have proposed a design for robot manipulators for use in retail stores but it is only able to move one object at a time. It is therefore thought that this task should still be carried out by human store clerks.

Task #3 There are already several robots on the market that detect misaligned products, for example Simbe Robotics’ Tally robot [BRON]. This robot gives staff members an overview of retail statistics, also alerting where products are misaligned. Staff members still have to do the task themselves, as there are no robots on the market that can perform this action.

Task #4 requires a way customers can interact with the robot. Cheng, C. H. et al. [18] have already proposed a solution for this by designing a graphic user interface where customers can choose from product pictures, so that the robot can guide them to the product’s location or provide product information. Moreover, the customer is also able to give the robot commands or ask questions through speech.

Task #5 requires the robot to detect the product that is given, and should then be told what replacement is needed. It should then be able to find both product’s locations and replace the products accordingly. This requires advanced sensors and actuators and this is probably not possible as of now, since current actuators are not capable enough.

Task #6 is possible through the design of a robot, however this design will differ a lot from a store clerk robot, so it should be seen as a different robot. This task might be accomplished with a forklift type robot that can do the basic task of unloading pallets from a storage truck. For low budget supermarkets where most products are presented in pallets rather than shelves this task is especially important, when now they can also perform the restocking task. The design of a pallet lifting robot is given by Garibotto, S. et al. [BRON] However, it should be noted that this design does not incorporate appropriate movement in environments with humans walking around.

Task #7 could be achieved through a similar approach as task #6 where pallets from the storage are brought into the supermarket so that store clerks can do their restocking tasks afterwards.

Survey

Link to the survey:


Most important design problems

From the literature study, the most important design difficulties that are faced currently will now be summarised and possible solutions will be given.

Robot navigation in a crowded supermarket environment

Navigating a supermarket is a problem of its own, systems for autonomous navigation exist but they do not have to take into account a chaotic environment like a supermarket. Our robot will have to be able to navigate around people and other obstructions. In order to handle this we need the robot to make a decision every time it encounters a possible obstacle. The main ways of dealing with an obstacle are:

1. Go around it. If there is enough space to go around the obstacle it can move around it. It should however take into account that it should not move too close to people as this might make some people uncomfortable (how close it should come to people may be answered by the survey) whereas there is no problem with it coming close to inanimate objects.

2. Request that the obstacle moves. If the obstacle is a person or belongs to a person, for example a shopping cart that somebody is holding the robot can request that the person moves. The request can be a spoken voice line or simply a sound and possibly things like blinking lights, the request can be made to be more explicit the longer the robot is waiting. When dealing with an inanimate obstacle it could also remotely notify other staff members to request that they move the obstacle.

3. Avoid it entirely. If the obstacle is not directly in front of the destination it should be possible to find a different route to the destination if the current one is obstructed, this method should be prioritized when dealing with people as they might be uncomfortable with the robot coming too close to them.

Assuming the robot knows the layout of the supermarket (represented by a graph where the edges are aisles and the vertices are the intersections) our navigation system should roughly follow the following protocol when tasked with navigating from point A to B:

- Find the shortest route from A to B in term of vertices (aisle intersections)

- Save the encountered obstructions in a graph

- While not at point B:

(

- Look through the aisle that leads to the next point and if there are no obstructions go through it.

- If there is an obstruction:

(

- Look through the aisle that leads to the next point and if there are no obstructions go through it.

- If it is an object:

(

- Notify staff that there is an obstruction

- Try to estimate if it can move around it

- Go around it if possible, otherwise:

- Try to find another way to point B, or otherwise:

- Wait

)

- If it is a person:

(

- Try to find another route to B, otherwise:

- See if there is room to go around the person without getting too close, otherwise:

- Wait in front of the person until it moves, possibly turn on a light signal, after waiting for some set amount of time ask them to move with a sound

)

)

When the store is represented by a graph a shortest path from point A to B can be found by simply using one of the well known algorithms for doing this like Dijkstra’s or A*. When searching a new path around an obstruction save the location of this obstruction in terms of an edge or vertex on the graph and search using the graph excluding the edges and vertices that have been marked as obstructions.

Shelf restocking according to the FIFO principle

Helping and interacting with customers

Shelf analysis providing shops with retail statistics

References

References, currently shown as [BRON] will be added in a later stage.

Appendices

Prototype Design (obsolete)

Schets:

https://drive.google.com/open?id=16Sy5iieEEjX_pO0cnagW_NCwODmixN-Z

Transport

Voor transport moet de robot natuurlijk uitgerust zijn met de benodigdheden om zich te kunnen verplaatsen. Hiervoor heb je natuurlijk energie en materiaal nodig om dit te kunnen realiseren. De energie wordt geleverd door middel van elektriciteit, in dit geval een batterij of een accu, aangezien de robot mobiel moet blijven. Als materiaal lijkt het het handigst om ouderwetse wielen te gebruiken met banden en natuurlijk een frame waar het aan vast zit.

Materiaal

De robot zal wel tegen een stootje moeten kunnen. Hiervoor moet het gemaakt zijn van sterk genoeg materiaal, maar moet daarnaast niet te zwaar zijn wat de verplaatsing belemmert. Als materiaal kunnen we denken aan aluminium met een stevige constructie.

Vorm: De robot zal voornamelijk cilindervormig zijn (R2D2-achtig)

Dimensies:

1,2 meter hoog, zodat men makkelijk er overheen kan kijken, zonder dat de robot niet opvalt. Touchscreen is daarnaast rond de hoogte van de middel van een gemiddeld persoon. Dit maakt op het scherm tikken gemakkelijk. 0,5 meter straal, de robot zal ongeveer 1 meter zijn in de breedte wat voldoende is om door paden te lopen en niet in de weg te staan voor klanten/medewerkers 10 cm Dikte, genoeg om tegen een stootje te kunnen, niet te zwaar te zijn en geen verspilling van materiaal Totaal volume: Oppervlakte grondvlak x hoogte = πr^2 x h = π x 0,5 x 1,2 =

  • Dit zijn allemaal schattingen

Materiaal kosten per kg: 1,7 euro/kg Bron: https://www.aluminiumprijs.eu/ Misschien nog een plastic/polymeren omhulsel zodat de robot minder “industrieel lijkt”

Kleur: Wit/Grijs, niet al te afleidend of intimiderend. Voornamelijk een “neutrale” kleur


Touchscreen

Een van de vormen van interactie met de robot zou kunnen via een touch screen met een vriendelijke user interface. Hiermee kunnen klanten of medewerkers kijken waar wat ligt in de winkel, wat op voorraad is of wanneer een nieuwe levering komt van een specifiek product. Kost ong: 50-200 euro

Bronnen:

https://www.nowthatsit.nl/lg-philips-lp140wh2-tft-scherm-14-0-1366x768-wxga-glossy-led.html?gclid=CjwKCAjw2_LcBRBYEiwA_XVBU5k0Km0YDs8uN0TA6DCZRFoE958pUCjlfXN7MFaDhG144Onb5cUtlBoCAiMQAvD_BwE https://www.distrelec.nl/en/tft-touchscreen-for-raspberry-pi-adafruit-2441-pitft/p/30091183?channel=b2c&price_gs=49.8641&source=googleps&ext_cid=shgooaqnlen-na&kw=%7Bkeyword%7D&gclid=CjwKCAjw2_LcBRBYEiwA_XVBU4rM3PFyQMzFjZEPnrCFvCgNF2owDs807ipUmvWwq28ipNFWP_woVhoCAQMQAvD_BwE

Barcodescanner

Van plaats constant te zoeken naar een barcodescanner in de winkel, kan je simpelweg de robot benaderen om te kijken hoeveel een product kost en enige overige informatie. Kost ong: 30-50 euro

Bronnen:

https://www.google.nl/search?q=barcode+scanner+kopen&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjPitW-zb3dAhWN2qQKHaCEDwwQ_AUICigB&biw=1536&bih=759

Voice Commands

Om met de robot te kunnen communiceren, zou het mogelijk zijn om dat spraaksgewijs te doen. Hiervoor heb je natuurlijk een microfoon nodig en een speaker voor de robot zijn output bijvoorbeeld. Daarnaast is een de juiste software ook vereist. De kosten hangen voornamelijk af van de prijs van de microfoon, aangezien software developen arbeid verricht. Dit kan zelf gedaan worden of ontwikkeld worden via outsourcing of gebruik maken van source code.

Kost ong: 2-100 euro

Bronnen: https://www.tinytronics.nl/shop/nl/sensoren/geluid/max9812-microfoon-versterker-module-met-microfoon https://www.google.nl/search?q=omnidirectional+microphone&source=lnms&tbm=shop&sa=X&ved=0ahUKEwiZw-bwhcDdAhVIK1AKHTFkAcYQ_AUICygC&biw=1536&bih=759 https://www.thomann.de/nl/samson_ub1.htm?glp=1

Sensoren

De robot heeft sensoren nodig om een pad te volgen, obstakels te herkennen en dingen op te pakken en neer te leggen. Dit zijn voornamelijk infrarood en ultrasone sensoren.

Kost ong: 400-800 euro

Bronnen:

https://www.roboteq.com/index.php/buy-dc-motor-controller-online/ordering-online?categoryTab=magnetic

Gripper: De gripper kan bestaan uit een kant en klare kit die op internet te vinden is, of we kunnen het handmatig maken door middel van losse gripper onderdelen en servo’s.

Kosten: 500-1500 Euro

Bronnen:

https://alexnld.com/product/diy-robotic-arm-expansion-pack-for-starter-robot-kit/?gclid=Cj0KCQjw_vfcBRDJARIsAJafEnFqTj3UQrSqz9o-L282qtoYznL-A94pOUS3XGhcpXZ2UF5LkoCGQxwaAsFREALw_wcB


Voorbeelden:

https://www.thinkbotsolutions.com/shop/mobile-industrial-robots-mir100 https://www.thinkbotsolutions.com/shop/mobile-industrial-robots-mirhook-100


Overige info:

https://www.youtube.com/watch?v=f0P1Ikyz8To https://www.mirror.co.uk/tech/robot-working-supermarket-fired-after-11892386 https://www.forbes.com/sites/parmyolson/2018/02/22/robotics-amazon-grocery-supermarkets/#7cf9d0b61e48 https://nypost.com/2017/02/05/inside-amazons-robot-run-supermarket-that-needs-just-3-human-workers/ https://www.twinkle.be/achtergrond/170141/hoe-robot-opmars-maakt-supermarkt/ https://www.voanews.com/a/robots-retail-supermarket-tech/4216442.html https://blog.robotiq.com/bid/52673/Pneumatic-robotic-grippers-vs-electric-grippers https://blog.robotiq.com/bid/58812/Robot-gripper-How-much-does-it-cost

Boodschappenlijstje

Arduino Uno Micro Controller

https://www.tinytronics.nl/shop/nl/arduino/main-boards/uno-r3-met-usb-kabel

https://www.tinytronics.nl/shop/nl/arduino/main-boards/uno-r3-compatible-usb-b Wielen

https://www.tinytronics.nl/shop/nl/arduino/kits/auto-kit-zelfbouw-rond

Gripper Kit

https://www.tinytronics.nl/shop/nl/robotica/toebehoren/servo-robot-grijper?search=grip

https://www.parallax.com/product/28202

Ultrasound sensor

https://www.tinytronics.nl/shop/nl/sensoren/afstand/ultrasonische-sensor-hc-sr04

Infrared sensor

https://www.tinytronics.nl/shop/nl/sensoren/optisch/optische-reflectie-sensor-infrarood-ir-950nm-tcrt5000?search=infra

https://www.tinytronics.nl/shop/nl/sensoren/optisch/vuur(infrarood)-sensor-module?search=infra

https://www.tinytronics.nl/shop/nl/sensoren/optisch/optische-reflectie-sensor-infrarood-ir-950nm-cny70?search=infra

Body kit/Frame

(Included in gripper or Wheel kit)

Servo’s

https://www.tinytronics.nl/shop/nl/robotica/motoren/motor/mg90s-mini-servo?search=servo

https://www.tinytronics.nl/shop/nl/robotica/motoren/motor/mg996r-servo?search=servo

https://www.tinytronics.nl/shop/nl/robotica/motoren/motor/s3003-servo?search=servo

Batterypack

https://www.tinytronics.nl/shop/nl/batterij-en-accu/batterijhouders/4x-aa-batterij-houder-met-losse-draden?search=batterij

Weerstanden

https://www.tinytronics.nl/shop/nl/componenten


Optioneel:

Encoder

https://www.parallax.com/product/28107

Controller

Camera

LCD scherm

https://www.tinytronics.nl/shop/nl/display

Microfoon

Speaker

Gyroscope

Prototype

Abilities: Keeping track and managing storage/stock Picking up goods from the storage Transporting the goods to the corresponding shelve(s) Putting the goods into the shelve(s) FIFO? (First in first out) Avoid a run in with a customer Communicate with customers when needed

User Requirements: Safe and efficient ground floor robot movement Users do not want to be obstructed by the robot’s movement, so the robot should have mechanisms to avoid obstacles and humans. It should move efficient to reach areas where tasks need to be completed as quickly as possible. Efficient refilling of shop shelves Shop owners strive for a high turnover, so it is important that shelves are in stock. Careful handling of shopping goods Inflicting damage to shopping goods will lead to having to throw them away, which is detrimental for the shop’s turnover and thus undesired for shop owners. Shop customers must be able to interact verbally with the robot Shop customers need to be able to gather information about various shop related topics Low cost maintenance of robot store clerk The robot should require less maintenance cost than the average salary of a human store clerk. Low error rate in detection of shopping goods The robot should correctly detect shopping goods with a low error rate for high customer satisfaction

Possible options for ground floor movement:

Simple wheels This speaks for itself, the movement is determined by a pair of wheels. This could be 2 (with a third one for stability) or 4 wheels. Caterpillar tracks Mostly used in the military on tanks or other specialized vehicles. Very convenient for getting around on rough terrain. In this case definitely not a priority but just mentioning it to keep it up for consideration. Legs Pretty ambitious, but could be used to make the robot more anthropomorphic/human like. Similar to companion robots and could be used for better interaction with customers. Stability is mostly the greatest obstacle to overcome when implementing this method.


Mapping: One idea we had was giving the robot the ability to map the area and keep track of its own position. This would be possible by programming the arduino to see the whole area as a grid that is divided by blocks, just like a graph. The starting point is the storage area, which in turn is the origin of the graph. Keeping track of it’s position would be possible by equipping the robot with compasses, that could calculate the angle in which the robot is currently in. This way, every path that the robot takes can be represented by vectors. This can come in handy when the robot has the whole map already predetermined in its programming.

Predetermined tracks (Black & White or magnetic strips): The floor of the supermarket kan have lines that are either simply black or magnetic. This way the robot can follow the tracks to get to a certain shelve with predetermined paths that it should follow which are the quickest ways to get to the right shelve. With simple black tracks, the robot can use infrared sensors to pick up the differences between black and white (or another lighter colour). And with magnetic tracks it could use a magnetic guide sensor. It would also be plausible to use the two methods in conjunction with each other to distinguish certain shelves by assigning certain colours on the tracks, so that it knows if it is the right shelve.

Possible sources: https://www.roboteq.com/index.php/applications/100-how-to/278-building-a-magnetic-track-guided-agv

Manual control The robot can be manually controlled by a human with the help of a joystick for example. This is the easiest way the control the robot with the least amount of programming, but less beneficial if we want to replace humans if there is human intervention of course.

Possible options for picking up goods: Gripper The most straightforward way of going about, it is a simple gripper that clamps itself onto the goods to pick it up and transport it to the desired location, where it afterwards softens its grip to put it down/into a shelf. Treadmill From the gym onto the supermarket scene, a treadmill could also be a way of picking up goods from underneath, instead of from the sides. Could be nice for more fragile goods, but not that diverse. Suction Just like a vacuum, it could suck goods onto a plateau. Afterwards station it in front of a shelve to then blow the goods into place. Probably not feasible in practice but still nice to think about. Scooping/lifting Picking goods up like a shovel or like the well-known forklift truck. Also an easy method to pick things up, but the goods could be tipped over because of the lack of support on the sides.

Possible options for avoiding collisions: Ultrasound sensor In order for the robot to detect any obstacles or customers in its way, it should have a method to detect them. A good way to do that, is the use of ultrasound sensors. With these sensors, the robot can send ultrasonic sound waves to an object within its vicinity which in turn makes the waves bounce back. These waves that are sent back to the robot can be picked up to know if there is an object in front. With the use of the speed of sound that the waves travel and some simple math, we are able to calculate the distance of the object with respect to the robot. Camera It is also possible to use a camera to detect an object in front of the robot. This is the easiest way to recognize something if the robot is manually controlled by a human, but this requires human cognition. The robot can not make out what is in front of it, because the camera detects anything, whether or not there is an object. This can be resolved by A.I. programming but that would probably be overkill, when there are better options to choose from.