PRE2022 3 Group12

Logbook

The logbook with member task and time spend per week can be found on the Logbook page

Problem Statement:

Society is currently faced with an ageing population. By around 2040, it is expected that one-quarter of the population will be aged 65 years or older. Compared to today, the size of this group of people will have increased by about 1.2 million people by 2040, all while the number of people working (in the age group 20 to 64 years old) will stay roughly the same. This means a large shortage of healthcare workers will arise, causing some elderly to not receive all care they might be expecting. One important aspect of this care that might easily be overlooked are ways to combat their loneliness. This is often prevalent among the elderly, especially those aged 75 years or older. One possible way to battle loneliness is to provide activities. However, with the reduced availability of care, it will become harder for healthcare workers to provide these activities. In these circumstances, robots can be used to support the workers.

Users:

Our design provides people with an opportunity to play physical card games without the need for other players. This is beneficial for anyone who is for some reason unable or uninclined to play with others. While it is great to have many potential people that are able to use the product, it also results in a large and ill-defined target group. In order to combat this general target group as well as form a starting point for the design and make it feasible considering the size of this project, a subset of the target group is taken. This new target group focuses on elderly people.

The target group of elderly people is chosen as they are generally assumed to have more difficulties with technology. It’s therefore expected that if the elderly people are able to properly use and understand the product, the younger generations will be able to do so as well.

We hope to increase the Quality of Life (QoL) of the elderly by creating this product. For example when they are unable to visit others, or unable to have visitors, they can still play with the robot and enjoy a game of cards.

User Requirements:

Due to their age, most elderly have increased problems with their sight, hearing, or motor skills. Therefore, it is important that the design has options built in to deal with this. For example, an easy-to-read font and text size, clear and loud audio implementations, and a lightweight and easy-to-move design.

Through our literary research, it was also noted that elderly people often experience more difficulties when learning something new. Because of this, it is assumed that using concepts that the elderly are already familiar with, or at least similar to those, is better as they will understand and learn them faster. Therefore, we should choose a game that is easy to understand and known by elderly people. As well as implement a simple interface and design.

Other aspects that could be added in order to improve the user experience, but are not necessary. Are the implementation of motivational messages during the game and multiple difficulty settings as a balance between ability and difficulty is important.

To engage the users’ more while playing with the robot, it is important that the robot has a competitive nature. Instead of having a robot that is relationship driven.

Deliverables:

The aim of this project is to design and build a card-game playing robot to provide social support in the form of entertainment for the increasingly ageing society. Therefore, it is the goal to deliver a prototype version of a card-game playing robot that satisfies some basic requirements that are necessary for the robot to be functional. The most important requirements are listed below.

1. The robot must have a strategy to play one specific card-game with equal performance as the average non-professional human player.
2. The robot must be able to successfully recognize and distinguish all the cards that make up a standard 52-card deck with an accuracy of at least 95%.
3. The robot must be embodied and it must be able to move the cards physically or have an integrated virtual environment by means of a multi-touch table.

In view of these requirements, the following components that make up the prototype will be delivered.

1. An algorithm that uses available information to select the next action or move.
2. An object recognition algorithm trained for a standard 52-card deck.
3. A physical implementation of the robot that is able to move cards or display them virtually.

The physical prototype will then be created by combining these components into one system. All progress of the project will be documented on this wiki, which will serve as the group report at the end of this project. Furthermore, a final presentation will be given at the end of the project combined with a demonstration of the prototype.

Approach:

Graphical representation of the steps of the design process.

During the project, the prototype will be developed by means of a design process. This process consists of multiple phases. Firstly, the problem will be defined as well as the design goal to solve the problem. Secondly, the functional and technical requirements of the prototype will be specified taking into account time, money and resources. From these requirements, it is possible to create concepts for the design. Next, details will be added to the design and the design will be realized. Lastly, tests will be performed to see if the prototype satisfies the requirements. This process can be summarized in the following steps:

1. Define problem
2. Specify requirements
3. Preliminary design
4. Detailed design
5. Prototyping
6. Testing

Given that the final presentations will be planned in week 8 of the course, this process will have a time period of seven weeks. Therefore, every step is expected to be completed within one week, with the exception of the detailed design since this is the most important and difficult step which will therefore require more time. So the time will be managed as follows, step 1 will be completed in week 1, step 2 in week 2, step 3 in week 3, step 4 in weeks 4 and 5, step 5 in week 6 and step 6 in week 7. In this way, all deliverables are finished in week 8.

Another important aspect of the design process is communication. Therefore, a weekly meeting is scheduled to discuss the progress and task division as well as a weekly work session. The meetings and work sessions will be scheduled on Wednesday from 10:30-11:30 and Monday 11:30-12:30 respectively.

Milestones:

The project milestones are both related to the steps of the design process and deliverables in that they break down the project into smaller sections for the tasks that need to be completed. They can be defined as follows:

1. Have a problem definition that clearly states the broader issue and the targeted user as well as the design goal
2. Have created a list of at least 5 requirements for both functional and technical requirements according to the MoSCoW method
3. Have created a complete mock-up (either digitally or physically) that clearly shows how all the requirements are satisfied
4. Have created an algorithm that implements a strategy to select actions for one specific card game
5. Have created an object recognition algorithm to identify a standard 52-card deck
6. Have a bill of materials
7. Have created a system to pick up playing cards
12. Have created a blueprint of the robot's physical embodiment
13. Successfully build, combine and integrate all components
14. The system satisfies the requirements as shown by means of a test

Each milestone will be assigned in a logical way to one step of the design process as shown by the following table.

Week	Step	Milestones
1	Define problem	1
2	Specify requirements	2
3	Preliminary design	3
4	Detailed design	6
5	Detailed design	3, 5, 7, 8
6	Prototyping	9
7	Testing	10

Task Division:

The design process consists of multiple phases and each phase conforms to different types of tasks. For example, the first phase includes a literature study, users study, problem statement and planning. Therefore, the tasks will be divided weekly among the group members during the team meetings. The task division for each week is shown in the tables below.

Week 1	Tasks
Abel Brasse	User study, study 5 scientific papers
Linda Geraets	User study, study 5 scientific papers
Luke van Dongen	Card recognition software, study 5 scientific papers
Sander van der Leek	Robot design sketch/model, study 5 scientific papers
Tom van Eemeren	Approach, milestones, deliverables, study 5 scientific papers
Tom van Liempd	Problem statement, study 5 scientific papers

Literature Study:

Card games and AI

Policy-Based Inference in Trick-Taking Card Games

Summary:

This paper describes how an opponent model is used for inference in trick-taking card games, like Contract Bridge, Skat, and Hearts. These card games introduce uncertainty by featuring a large amount of private information, which becomes known after a long sequence of actions. Therefore, the number of histories is exponentially large in the action sequence length and extremely large information sets get created.

Deterministic search algorithms such as Perfect Information Monte Carlo and Information Set Monte Carlo Tree Search have been employed. However, due to non-locality, deterministic search has been heavily criticized. To deal with these issues,  inference helps by biasing state samples so that they are more realistic with respect to the opponent’s actions. Therefore, inference is a central concept in imperfect information games and plays a key role in the performance of deterministic search algorithms. This involves an opponent model to determine unknown information based on the action sequence.

The paper describes how policy-based inference was employed for Skat, which is a 3-player trick-taking card game and is played using a 32-card deck where cards 2 through 6 from each suit are removed from the standard 52-card deck. The actions are determined by three factors each having a certain probability. Firstly,  the world relates to chance nodes in dealing and can be directly computed. Secondly, our own actions with a probability of 1 since we choose actions leading to a given state with full knowledge of our strategy. Lastly, a given state within the information set due to other players’ actions can only be determined perfectly if we have access to the other players’ policies. However, there are two issues. Firstly, we do not have access to the other players’ policies or they are computationally too expensive to model. Secondly, the number of states in the information set can be quite large. For these reasons, the authors suggest sampling the worlds and normalizing the distribution over the subset of states.

The article concludes that policy-based Inference appears to provide much stronger inference than other methods such as Kermit Inference and Card Location Inference. Furthermore, the authors conclude that sampling card configurations are more effective than sampling states. Lastly, it is suggested to experiment with heuristics that allow the algorithm to find states that are highly unlikely and discard them to improve the performance.

Reference:

D. Rebstock, C. Solinas, M. Buro and N. R. Sturtevant, "Policy Based Inference in Trick-Taking Card Games," 2019 IEEE Conference on Games (CoG), London, UK, 2019, pp. 1-8, doi: 10.1109/CIG.2019.8848029.

A Social Robot as a Card Game Player

Summary:

In this paper, it is investigated how a social robot player that is able to play the card game with social behaviours towards its partner and its opponents can be built. The authors state that generally, social robots can contribute with new ways of creating socially engaging interactions with humans in entertainment contexts.  For instance, physical embodiment can provide a more immersive user experience, an improved game feedback and a more believable social interaction. However, for multi-player games played in the physical world, the social environment becomes even more relevant. Therefore, the paper finds an answer to the question of how people will perceive a social robot player compared to human standards and if people are willing to trust a social robot to be their partner in a team game.

For their research, the authors employ a social robot, which is able to express emotions, provide spoken feedback, and respond socially, for the team card game called Sueca, which is a non-deterministic game and is considered an imperfect information game. Furthermore, this paper explores how the algorithm’s parametrizations affect the performance-time configuration.

The robot has two modules, the game module responsible for choosing moves and the social module responsible for social behaviour. The game module evolved over three stages. Firstly, a rule-based player was created that replicates the general gameplay strategy of non-professional human players. The performance of this agent was found to be similar to the performance of human players. Secondly, the Perfect Information Monte-Carlo (PIMC) algorithm was applied, which is a suitable algorithm for partially observable environments. With PMIC, the hidden information is sampled several times and the best move is computed by solving exactly or heuristically in each perfect information game by use of the MinMax algorithm. However, PMIC has two disadvantages, namely strategy fusion and non-locality. On top of that, success of PMIC depends on three game properties, namely leaf correlation, bias and disambiguation factor. Thirdly, a hybrid approach was employed to respect the time constraint without bounding the depth of the search. Here PIMC was only used from a certain tick on. Up to that tick, a stochastic version of the rule-based strategy was adopted.

Additionally, the social module ensures that the robotic player engages in social interactions using both verbal and non-verbal behaviours. The emotional and social behaviours of the robot were built by An emotional agent framework (FAtiMA) allowing the robot to not only play competitively but also respond emotionally in a natural manner by emotional appraisal, social and emotional behaviours.

The experiment was performed by using the autonomous robot EMotive headY System (EMYS) over a multi-touch table using physical cards. Furthermore, the approach was tested by a user study to compare the levels of trust that participants attributed to the robot. The results showed that human players increased their trust in the robot as their game partners. Furthermore, results have shown that the robot team had a winning rate of 60%.

The authors conclude that the social robot showed similar results in terms of trust from a human partner as humans partners,  indicating that we trust a robot to be our game partner in a card game. This also shows the success of the social human-robot interaction.

Reference:

Correia, F., Alves-Oliveira, P., Ribeiro, T., Melo, F., & Paiva, A. (2021). A Social Robot as a Card Game Player. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 13(1), 23-29. https://doi.org/10.1609/aiide.v13i1.12936

Exploring the Entertainment Value of Playing Games with a Humanoid Robot

Summary:

In this paper, a social robot plays the Mastermind game with a human to study how behavioural patterns of the robot affect the entertainment experience of a human player, especially when the behavioral patterns are tied to events in the game in a meaningful way. A major societal challenge is the ageing society due to an increased life expectancy as this decreases the support ratio resulting in a high pressure on caregivers. The authors mention three reasons why a social robot could provide social support in the form of entertainment. Firstly, studies have shown that this improves elderly people’s well-being significantly. Furthermore, an entertaining robot could engage its users in other tasks such as health exercises, taking their medicine on a regular basis or measuring health functions like blood pressure. Research has also shown that physical embodiment is more entertaining than playing against a screen-based character since it offers more diverse, richer forms of interaction.

The results showed that the robot’s behavior did not seem to affect the participants’ enjoyment of the game. However, the authors conclude that people experience playing games with a robot as entertaining. Furthermore, by combining speech, gestures, and eye LED patterns, robots can imitate very subtle levels of emotions that are correctly perceived by humans.

Reference:

Johnson, D.O., Cuijpers, R.H., Pollmann, K. et al. Exploring the Entertainment Value of Playing Games with a Humanoid Robot. Int J of Soc Robotics 8, 247–269 (2016). https://doi.org/10.1007/s12369-015-0331-x

Understanding the Success of Perfect Information Monte Carlo Sampling in Game Tree Search

Summary

In this paper, it is shown how it can be decided from game properties whether Perfect Information Monte Carlo (PIMC) for playing imperect information games that are too large to be optimally solved will be an effective approach to new and unexplored games. PIMC makes two types of errors, namely through the concepts of strategy fusion and non-locality. On top of that, it depends on the properties leaf correlation, bias and disambiguation factor of the game.

Reference

Long, J.R., Sturtevant, N.R., Buro, M., & Furtak, T. (2010). Understanding the Success of Perfect Information Monte Carlo Sampling in Game Tree Search. Proceedings of the AAAI Conference on Artificial Intelligence.

Object recognition

Real-Time Detection and Recognition of Cards in the Game of Set

In this paper a procedure is described for detecting and recognizing which cards of the game Set is visible in a real-time video feed. Object detection methods are used to detect the presence of cards in a video sequence and to recognize which cards of the game of Set are present by using the different features of the cards. The procedure consists of three steps. Firstly, the individual cards get extracted/segmented from the images of the real-time video feed. Secondly, a deep two-layered fully connected convolutional network performing feature extraction is used to separate each image. Finally, valid set combinations are found using feature interference in order to assign an image of a card to the correct set.

In this paper, it was the aim to recognize which cards were present as fast as possible. Hence, the card detection, feature extraction, and classification all needed to be performed fast. This had its consequences. In order to have fast feature extraction only a limited number of pixels of each image were used, making the feature extraction procedure less robust.

In this paper, cards of the game of Set were used. These cards have four different features: shading (solid, striped, or outline), shape (oval, squiggle, or diamond), color (red, purple, or green) and number of shapes (one, two, or three). For every card feature one classifier was trained. In addition, another classifier was trained to determine whether the segmented part of the image was a card of the game of Set at all. So, in order to train the classifiers, the representative dataset also consisted of some cards which were not part of the game of Set.

In the procedure of extracting the cards from the images of the real-time video, the OpenCV framework is used. OpenCV uses contour finding on grayscale images which separates the cards, which should be white, from the background, which should be black, by using grayscale images.

Reference:

Ditrih, H., Grgić, S., & Turković, L. (2021, September). Real-Time Detection and Recognition of Cards in the Game of Set. In 2021 International Symposium ELMAR (pp. 161-164). IEEE.

Pothole Detection using CNN and YOLO v7 Algorithm

In this paper, the YOLO v7 algorithm together with the Google API and an accelerometer is implemented in an app in order to identify and locate potholes. The YOLO v7 algorithm is used in order to categorize the potholes and the Google API and Accelerometer are used for counting the potholes. At the time, the YOLO v7 algorithm was the most recent version of the algorithm. The newest version, YOLO v8, has been released recently. However, not much work has been released using this version yet. Hence, this paper has been chosen.

The YOLO, which stands for ”You Just Look Once”, paradigm states that you only need to look once in order to recognize an object. YOLO v7 is a real-time object detector which has started a revolution in the way computer vision is practiced. With YOLO you can make accurate predictions about locations, classes and bounding boxes using visual frames, for  instance the frames of a (real-time) video. According to this paper, the YOLO algorithm is considered to be one of the most efficient methods for locating potholes, as it is both accurate and quick.

Considering the statements made about the effectiveness of the YOLO v7 algorithm for object detection in this paper, using this algorithm in our project might be a good idea.

Reference:

Reddy, E. S. T. K., & Rajaram, V. (2022, December). Pothole Detection using CNN and YOLO v7 Algorithm. In 2022 6th International Conference on Electronics, Communication and Aerospace Technology (pp. 1255-1260). IEEE.

Implementing Playing Cards BlackJack Game using OpenCV

In this paper, OpenCV is used to create software which detects and recognizes playing cards and decides which step is the best next step to perform in a game of Blackjack.

The OpenCV library is an open source computer vision and machine learning software library built to give a structure for computer vision applications. This library can be used for  recognizing faces, identifying object and much more. OpenCV is compatible with mostly every device and is provides many API’s.

In the literature study of this paper, a group of developers is discussed who used computer vision to identify playing cards during a poker game. The developers described a series of steps in their procedure of identifying playing cards Firstly, they extracted the card from the playing field by the contrast between the playing cards and the field they are resting on. Next, they used “template matching” to identify which cards are on the playing field. The program used by the developers uses probabilities to identify the card that it sees. This is possible due to the fact that a deck of playing cards has a finite number of possibilities, as there are only 52 cards in a deck.

In this paper, some challenges that needed to be faced in order to recognize cards successfully have been described. The lighting conditions should be good, because the definition of objects is influenced by lighting and bad lighting might influence the categorization of objects negatively. The background should be contrasting the objects as much as possible in order to prevent objects blending into the backdrop, making identification difficult. Objects might be only visible from a specific angle/viewpoint, this can make an object appear differently and hence influence the categorization of the object negatively. These challenges should be taken into account when performing object recognition.

Reference:

Mohd, T. K., Akkar, A., Cassens, J., Cregan, S., & Vander-Pallen, M. A. (2023). Implementing Playing Cards BlackJack Game using OpenCV.

Playing Card Recognition Using Rotational Invariant Template Matching

In this paper, a method for recognizing playing cards is described. In this paper, old playing card recognition methods have been extended such that cards that are rotated and/or have a different distance towards the camera also can be recognized successfully. By considering rotation and scaling robust playing card recognition can be obtained.

In this paper, a web camera is used in detecting the playing card. The camera acts as a sensor to scan the playing cards. It is found that the resolution of the camera is important. The higher resolution camera is used, the easier it is to achieve good results.

The playing cards are differentiated from the background by thresholding grey image. In addition, edge detection is used not only to find the edges of the cards, but also in order to find the contours of rank and suit.

A card can be recognized while being in any position in the background, only when rotation and scaling are considered in the process of card recognition. Hence, when doing the comparison between the obtained image and the card templates, the cards get rotated to the same position as the templates of the cards.

In the method used in the paper, character segmentation differentiates between card rank and card suit segmentation. Template matching is used on the segments and based on the degree of match it is decided which card is most likely in front of the camera.

In the paper, it is concluded that the same cards that are used to generate the templates need to be used when using the system, because the system cannot be successfully generalized to a wider variety of playing card decks when it is not trained on that.

Reference:

Zheng, C., & Green, R. (2007, December). Playing card recognition using rotational invariant template matching. In Proc. of Image and Vision Computing New Zealand (pp. 276-281).

Introducing Computers to Blackjack: Implementation of a Card Recognition System using Computer Vision Techniques

In this paper, a project is described in which a computer program is written which is capable of recognizing all cards of a standard deck of playing cards.

The playing card recognition procedure consists of multiple different components. Image thresholding was performed to differentiate between the cards and the black background by using a simple intensity histogram in RGB space. Grassfire transform was used to close up black regions in the cards. The grassfire transform was chosen because it is quick and simple. Then, a two-pass segmentation was used on the binary image to  segment out the different card regions and to find out how many cards were present in the image. Subsequently, card rotation was used to rotate the cards. In order to be able to perform card rotation data about the central moment of each card needs to be obtained and used. After that, a card thresholding process was used to process each card individually and the color of the card was recognized by producing an average intensity. Finally, after successfully producing the region maps of each card, both the suit and the number of each card was determined. Three simple networks were trained to distinguish between the three face cards.

In the paper, it is concluded that noise in the camera was the source of many misclassifications. Hence, the developers of the computer program implemented a voting scheme which runs the program for five frames before making a decision for each card. This design decision was made trying to eliminate the effect of the camera noise.

Reference:

Hollinger, G., Ward, N., & Everbach, E. C. (2003). Introducing computers to blackjack: Implementation of a card recognition system using computer vision techniques. Colby College, Waterville.

Poker card recognition with computer vision methods

The paper describes a method to recognize playing cards that does not make use of machine learning. Instead the paper describes the steps that can be taken to process an image such that the type of card can be determined based on mathematical properties.

An image of a set of cards that someone is holding, which is taken by a regular mobile phone can be processed using the following steps:

1. The RGB color-image is transformed into a grayscale image to reduce the amount of data, but retain the shapes that will be used later on for the detection.
2. Images often contain random noise, that might either be caused by the phone taking it or by the environment. To denoise the image, Gaussian filtering is used.
3. The grayscale image, with pixel values between 0 and 255, is binarized, meaning that each value is either black or white. This is done using a simple threshold that is picked based on the image.
4. The moment of the binary image is analyzed to get the region of the image containing the playing card. Moments are ways that describe the (geometric) features of an image.
5. Using Hu moments, finally the shape and symbols on the playing cards are recognized.

By applying above steps to known images, templates can be created that can be used later to detect actual playing cards.

Reference:

Hu, X., Yu, T., Wan, K., & Yuan, J. (2021, August). Poker card recognition with computer vision methods. In 2021 IEEE International Conference on Electronic Technology, Communication and Information (ICETCI) (pp. 11-15). IEEE. https://ieeexplore.ieee.org/document/9563607

Poker Watcher: Playing Card Detection Based on EfficientDet and Sandglass Block

The paper describes a way to detect playing cards from a camera feed using deep learning methods. In particular it is noted that for the available data, the cards can be quite small, which is why special care must be taken to achieve proper recognition.

The paper uses a neural network called EfficientDet, which is a single-shot neural network. There are also two-shot object detection algorithms, but these are less time-efficient and hence less suitable for real-time applications. The algorithm was specifically tweaked by changing certain blocks which it consists of by Sandglass blocks. These blocks are better able to preserve the high-dimensional representations of the images that are being recognized.

Lastly, the paper discusses methods of preprocessing. Because it is unknown from what angle the camera will be recording, steps are taken to transform the image such that it always creates a top-down view. This is achieved using a perspective projection.

Reference:

Chen, Q., Rigall, E., Wang, X., Fan, H., & Dong, J. (2020, December). Poker Watcher: Playing Card Detection Based on EfficientDet and Sandglass Block. In 2020 11th International Conference on Awareness Science and Technology (iCAST) (pp. 1-6). IEEE. https://ieeexplore.ieee.org/document/9319468

Object Detection in 20 Years: A Survey

The paper describes a few milestones made in the past 20 years for object detection:

1. Between 1990 and 2010, most object detection algorithms were built based on handcrafted features. Basic detectors used techniques like sliding windows and feature selection.
2. From 2012, object detection took a leap forward by using Convolutional Neural Networks (CNNs), which can be split up into two groups:
   1. Two-stage detectors make use of a “course-to-fine” process.
   2. Single-stage detectors operate using a single complete step.

A lot of different neural networks have been developed in the past decade, each one striving for a better accuracy and speed. Two-stage detectors over time have proven to give a higher accuracy, but are significantly slower than single-stage detectors.

A lot of advancements have been made over the years. A particular advancement that is worthy to note is that of lightweight network design. Some neural networks have been developed, taking in mind that they can be ran on lightweight edge devices: devices that might be placed inside a self-driving car or smart camera. These devices do not have a lot of computing power and hence are limited in the size of the neural network they can efficiently run. Neural networks can take this into account, by reducing the number of channels or parameters that each layer has to deal with, which makes it run faster, at the loss of accuracy.

Reference:

Zou, Z., Chen, K., Shi, Z., Guo, Y., & Ye, J. (2023). Object detection in 20 years: A survey. Proceedings of the IEEE. https://ieeexplore.ieee.org/document/10028728

A survey of modern deep learning based object detection models

This paper provides an overview of the popular neural networks that have been developed in the past decade. Some datasets are described which are commonly used to measure the performance of the neural networks. These datasets provide a large number of pre-labeled images which can be analyzed by the neural network to retrieve an accuracy score.

Similarly to the paper ‘Object Detection in 20 Years: A Survey’, some early non-neural-network-based object detection is described, as well as a number of two-stage detectors. Lastly, the single stage detector algorithms are described, which are most applicable to this project:

• YOLO is a single stage detector inspired from the GoogLeNet model for image classification. It surpassed other single stage models that existed at the time in both accuracy and speed. However, it had significant shortcomings in terms of localization accuracy for small or clustered objects. It also was limited in detecting multiple objects close together.
• SSD (Single Shot MultiBox Detector) was the first single stage detector to match the accuracy of other two-stage networks, which were known for better accuracy. It was also faster and more accurate than YOLO, but had similar problems detecting small objects.
• YOLOv2 was an improved version of YOLO, replacing large parts of the internal neural network structure. It had fewer internal parameters and hence offered better detection speed.
• RetinaNet is simple to train, converges faster and easy to implement. It achieved better performance in accuracy and run time than the two stage detectors. RetinaNet also pushed the envelope in advancing the ways object detectors are optimized by the introduction of a new loss function.
• YOLOv3 was another improved version of its predecessors, although no ground breaking changes were made.
• CenterNet brought a fresh perspective by not detecting objects using bounding boxes like other algorithms, but instead detecting them as single points. It is more accurate than the previously mentioned algorithms.
• EfficientDet (based on the EfficientNet backbone) is a scalable detector, which can scale dynamically based on the number of features. It again proved to have better efficiency and accuracy compared to it’s predecessors.
• YOLOv4 incorporated a lot of exciting ideas to design a fast and easy to train object detector that could work in existing production systems. It utilized a number of methods that increased the training time, but did not affect the inference time.
• YOLOv5 was another iteration of the YOLO algorithm, published by a company called Ultralytics. It was only released as a GitHub repository, and not as a peer-reviewed research, which caused doubts about the authenticity and effectiveness. However since it’s release, it has been utilized in several applications with effective results and has started to generate credibility.

Reference:

Zaidi, S. S. A., Ansari, M. S., Aslam, A., Kanwal, N., Asghar, M., & Lee, B. (2022). A survey of modern deep learning based object detection models. Digital Signal Processing, 103514. https://doi.org/10.1016/j.dsp.2022.103514

A Review of YOLO Algorithm Developments

The paper describes the many iterations the YOLO algorithm has gone through in the past years. It describes that the YOLO and YOLOv2 were not effective at detecting small targets. Among with other improvements, these issues were largely addressed with the release of YOLOv3. The newest version of YOLO (at the time the paper was written) was YOLOv5. This version of the neural network was much more flexible, and could easily be trained on your own dataset.

Reference:

Jiang, P., Ergu, D., Liu, F., Cai, Y., & Ma, B. (2022). A Review of Yolo algorithm developments. Procedia Computer Science, 199, 1066-1073. https://doi.org/10.1016/j.procs.2022.01.135

Problem, users and existing products

Designing social games for children and older adults: Two related case studies

Summary:

This article focuses on making a new board game, more engaging by adding technology. While focusing on two groups, namely children and elderly. The reasoning for combining the paper games and technology is, is that paper games have two important elements which games on computers either don’t have, or have less. So would the combination be able to still have the tangible parts, such as moving an actual piece across the board. As well as that the combination has more social interaction between players, as they can actually look each other in the face.

They based the gameboard versions for the children and elderly on what they usually play and their own suggestions. For the elderly they noticed that bank, poker and Black Jack were popular card games.

With the technology the playboard becomes more dynamic, sounds are added to create suspense for example, and the technology adds a bit of uncertainty to game. Both groups appreciated the uncertainty element, and found the version with the technology to be more engaging than the same paper version.

Reference:

A. A. Mahmud, O. Mubin, S. Shahid, J. Martens (2010). “Designing social games for children and older adults: Two related case studies”. https://doi.org/10.1016/j.entcom.2010.09.001

Friends or Foes? Socioemotional Support and Gaze Behaviors in Mixed Groups of Humans and Robots

Summary:

The study tries to figure out how humans and robots behave when interacting in small groups, with both humans and robots. And whether this could give new insights to how social robots should be designed. One robot that is used in the experiments is relationship driven and cooperative, namely Glin+. Whereas the other robot, Emys-, is competitive. For the experiment they focused on two elements, the eye gaze and the socioemotional support.

They concluded that participants looked more often at Glin+ when it was their partner, while they looked more at Emys- when it was their opponent. This could possibly be, because they consider Emys- to be more of a “threat” for their goal in the game, as Emys- is more competitive.

They also noticed that Emys- got more support when Emys- was a opponent, than Glin+ got when Glin+ was an opponent. This came as a surprise towards the authors, as they had expected that Emys- competitive behavior would have lead towards rivalry and would have undermined pro-social motivation. They suspect this might be, because the participants possibly wanted to appease Emys-.

The authors’ last conclusion is that partners supported each other more, and that the participants showed more support towards other humans.

Reference:

R. Oliveira, P. Arriaga, P. Alves-Oliveira, F. Correia, S. Petisca, A. Paiva (2018). “Friends or Foes? Socioemotional Support and Gaze Behaviors in Mixed Groups of Humans and Robots”. https://ieeexplore.ieee.org/document/9473499

Magic iCub: A Humanoid Robot Autonomously Catching Your Lies in a Card Game

Summary:

The article tries to create a robot that is autonomous, and is not driven with a Wizard of Oz approach. They try to make it autonomous by making the robots decisions based on the robot trying to measure a human’s inner state, they do this with eye tracking. Another goals of the robot is to do an entertaining activity with a human. For the game, the robot tries to guess which card is the secret card out of the 6 cards the human is describing. The human will pick 6 random cards, and pick 1 random card of the 6 to lie about when describing the cards. The card the human is lying about is the ‘secret card’. The cards don’t have any QR-codes, to reassure the human players that the robot could not cheat.

One of the problems that the robot has, is that the robot is sensitive to light-level changes. Although this is mainly an issue for outdoors, as light changes should not happen indoors.

In their conclusion they state that the robot is successfully able to measure a human’s inner state, as the robot has a high accuracy for guessing the correct secret card.  And that the robot is able to autonomously guide a human-robot interaction, as the measures of fun confirm that the game is entertaining.

Reference:

D. Pasquali, J. Gonzalez-Billandon, F. Rea, G. Sandini, A. Alessandra Sciutti (2021) . “Magic iCub: A Humanoid Robot Autonomously Catching Your Lies in a Card Game”. https://doi.org/10.1145/3434073.3444682

Just follow the suit! Trust in Human-Robot Interactions during Card Game Playing

Summary:

The paper’s aim was to create a social robot and an entertaining activity to reconnect the elderly, who often suffer from social isolation. They also hoped that it would contribute to the QoL elderly experience.

For their project they chose one of the most played games among elderly in Portugal, namely Sueca. Since robots become more competent, humans might start seeing them as fierce competitors. But people might still be ware of robots. Therefore, the paper tries to figure out how trust levels could work between humans and robots.

The authors concluded that humans do trust robots. But the level of trust they have in the robot, depends on previous encounters with the robot. Therefore, suggesting that to increase the trust level there has to be a longer period of being acquainted.

Reference:  

F. Correia, P. Alves-Oliveira, N. Maia, T. Ribeiro, S. Petisca, F. S. Melo, A. Paiva (2016). “Just follow the suit! Trust in Human-Robot Interactions during Card Game Playing”. 10.1109/ROMAN.2016.7745165

Motivational Factors for Mobile Serious Games for Elderly Users

Summary:

This paper focuses on understanding what motivates elderly users to use (serious) games. Serious games refer to game whose goal is not just amusement, so for example games that have also goal teaching the player something new.  The primary reason, users use a game is the usability of it. So games have to be adjusted to the needs of the elderly, such that it is easy to use. Elderly can namely have trouble with sight, hearing, attention, motor skills and using technology. Other motivations to use a game are that it is fun, relaxing, social interaction, gives regularly motivational messages, a good balance between ability and difficulty, and personalization of levels and time. Elderly also play games as a way to escape reality.

Reference:

R. N. S. de Carvalho, L. Ishitani (2012) “Motivational Factors for Mobile Serious Games for Elderly Users”. https://www.researchgate.net/publication/277709317_Motivational_Factors_for_Mobile_Serious_Games_for_Elderly_Users

Designing and Evaluating the Tabletop Game Experience for Senior Citizens

gaming is widely experienced as a means for social interaction. Especially for elderly with less leisure time. Elderly mostly play low-tech games such as card games. They are less inclined towards computer based games, this might be because they are unfamiliar with the newer games and more afraid of the complexity of both installation and playing.

Another point might be that games do not aim at elderly people and thus make games that are unfamiliar and hard to understand.

According to their field-research at an local community center elderly mostly play card games (e.g., bank, poker), guessing games and memory games. The majority of the games seemed as having simple and uncomplicated rules. Most elderly also moved between groups of different people as they play different games, most of which revolved around the tables.

From interviews it seemed that their game play was mostly of social nature, and that they disliked playing for money and avoided gambling. Primary motivation to play games is to have fun and to widen their social network.

al Mahmud, A., Mubin, O., Shahid, S., & Martens, J.-B. (2008). Designing and Evaluating the Tabletop Game Experience for Senior Citizens. https://dl.acm.org/doi/pdf/10.1145/1463160.1463205

A Single-User Tabletop Card Game System for Older Persons: General Lessons Learned From an In-Situ Study

This paper is about a study on the use of tabletop systems for card playing aimed at senior citizen with little to no computer experience. Researched whether the use of an alternative form of Briscola (card game) could be offered to older people who were (temporarily) restricted to their homes, or for some reason were unable or uninclined to play with others. The study was done in a home where participants often used the system with friends resulting in less information about isolated use.

They found that their BriscolaTable was used even though they could choose freely between other activities at the center, total of 67 games from 22 players in 2 weeks.

improvements could be made by paying more attention to motor limitations associated with age, specifically the drag and drop on a screen and the deterioration of touch in older adults.

They assume the ease of use that their participants experienced was mostly due to the correspondence between actions of the Table and normal card-playing. But as in general literature (according to this article) the consensus that it is unnecessary to stick rigidly to those metaphors, they found that they could have done more e.g. to include more ‘shortcuts’ (deal cards button, instead of manually dealing all cards 1 by 1), while also mentioning that some users would prefer to do some physical actions by themselves.

The study also found that the elderly found the agent that generally only canned utterances to be entertaining (even though not actually very useful). Probably also due to their ‘spectating friends’.

Gabrielli, S., Bellutti, S., Jameson, A., Leonardi, C., & Zancanaro, M. (2008). A single-user tabletop card game system for older persons: General lessons learned from an in-situ study. 2008 IEEE International Workshop on Horizontal Interactive Human Computer System, TABLETOP 2008, 85–88. https://doi.org/10.1109/TABLETOP.2008.4660188

How older people account for their experiences with interactive technology

This paper aims to complement existing work with a discussion of how people who are just starting to use computing technology account for the difficulties they encounter. This because the problems are not confined to physical and cognitive factors; attitude, anxiety, perceived relevance of the technology to everyday life, usefulness, usability,….

This was done as study sessions where older people learned to work with a personal computer, where they discussed the problems they encountered.

Reports of bad experiences; unpredictability of individual features, to frustration at ones own inability to remember necessary sequences of operations. Most of the bad experiences seem to relate to the perceived uncontrollability. But some themes were found among them; alienation (‘This is not my world at all’), identity (‘I worked in a job with people, not with machines’), agency (‘But sometimes you’re obliged to’), anxiety (‘I was frightened to’), age related (‘being too old’), being too busy (‘You haven’t got a space in the day to learn’), finding a purpose for the technology (‘I see their uses but I don’t have to accept them fully’).

Turner, P., Turner, S., & van de Walle, G. (2007). How older people account for their experiences with interactive technology. Behaviour and Information Technology, 26(4), 287–296. https://doi.org/10.1080/01449290601173499

It’s Food Fight! Designing the Chef’s Hat Card Game for Affective-Aware HRI

This paper describes the design of an interactive game between humans and a robot that makes it possible to observe, analyze, and model competitive strategies and affective interactions with the aim to dynamically generate appropriate responses or initiations of a robot.

The requirements they set for The Chef’s hat game were;

1.    The game should elicit a multitude of affected behaviors that the robot can properly understand and model.

2.    The game should be playable without the need of verbal expressions.

3.    The game should provide the possibilities of strategies based on bonding between players.

4.    The game should have specific mechanics that cause affective reactions.

5.    The game should be easy to follow and understand, clear turns and small number of actions.

6.    The game status should be easy to track and monitor.

7.    The game should give players enough opportunities to interact with other players (through game-mechanics).

Their design used a top-view camera that is used to monitor the entire game state, that way the robots can make decisions by analyzing a single image. The playing cards they used were given QR-codes so the robot could easily recognize them.

The goal for the robot involved in the game will be to understand the affective behavior of the human participants and use this information, together with that of the game status, and to select the best strategy to play the game.

This study also mentions the quality of the robots strategy, having it make accurate predictions for some (5) steps into the future could make it win with high likelihood. This would not make for an enjoyable gaming experience, the same can be said when the robot tries to let the player win al the time. To combat this they use an adaptive system that implements a strategy based on the emotional state of the other players.

Barros, P., Sciutti, A., Bloem, A. C., Hootsmans, I. M., Opheij, L. M., Toebosch, R. H. A., Barakova, E., & Toebosch, R. H. (2021). It’s Food Fight! Designing the Chef’s Hat Card Game for Affective-Aware HRI. https://doi.org/10.1145/3434074.3447227

Robot Design

Design of a Card Picking Robot

The aim of this paper is to explain how they designed a robot arm which is able to pick up cards from a stack and move them under a camera-scanner. They explain 5 candidate designs and discuss their benefits and pitfalls.

In all of their designs they use a suction cup which is connected to an air pump with a solenoid valve to create and release a vacuum on demand.

1. The first design was a 'plotter' robot (robot arm on a stationary frame). This design was easier to build than other designs since they owned an existing plotter but suffered from slow speeds and imprecise movement.

1. The second design was an 'air cylinder' robot, this design featured a stationary robot arm which can rotate on its axis and move the suction cup up or down (2-DOF). The main feature of this design is that all movement was achieved using compressed air.
2. The design scored high on precision but the cost was prohibitively expensive.
3. The third design was the 'rack and pinion' robot, similarly to an overhead crane this design has a stationary frame hanging above which guides the arm on a belt. But this design was deemed too difficult to manufacture.
4. The fourth design was the 'photocopier' robot. Similarly to a printer device all of the cards would be held in a magazine under some roller wheels and would be moved downhill to the scanner. This design was reliable but too costly to manufacture. (Besides this design would not fit our card-playing specifications since we need 2-way movement.)
5. The fifth design was the 'stepper motor' robot, this design uses a central shaft which can rotate the entire arm using a precise stepper motor, the arm can move up and down using a lever attached to a solenoid. This design was precise and affordable using mostly commercially available parts. After comparing the scores they decided that this stepper motor design was deemed the overall best and they decide to use this in their final design.

Diagram of their final 'stepper motor' design.

They use an innovative system which they call The Vibrabox. This uses a steady air-flow over a stack of cards which makes the top card float a few millimetres above the other cards, which makes sure that exactly 1 card is picked up at a time when grabbed by the suction cup.

The paper further gives a detailed list of sub-components, diagrams of how the inside of the robot looked like, and their robot controller code.

We will also require a system to read and classify playing cards just like their analysis plate, so we will keep a close eye on this design.

Overall this last design is nearly a perfect match to our needs in this project. We will adapt this design further to specialise it for playing a card game.

Bunyan-Ngwiri, D. (1992). Design of a Card Picking Robot. [1]

Multi-DOF counterbalance mechanism for a service robot arm

This paper discusses how to produce a a low-cost robot arm for any desired DOF using springs and wires to counterbalance the movement and reduce torque.

Each joint is compactly fitted with the counterbalance mechanism and a separate actuator.

This allows the robot to be powered by low-power/cheaper actuators since they need to fight the robot's inertia less.

The caveat of this design is that the 'payload' of the arm is limited by the strength of the actuators, but since we plan on moving only lightweight cards this shouldn't be an issue.

This design could be viable to replace the 'stepper motor' arm of the above paper if we require a higher DOF robot.

Kim, H. S., & Song, J. B. (2014). Multi-DOF counterbalance mechanism for a service robot arm. IEEE/ASME Transactions on Mechatronics, 19(6), 1756-1763.

Design of a 3 DoF robotic arm

This short paper gives a mathematical analysis of the inverse kinematics of a 3-DOF robot arm.

This can be useful if we decide to use a 3-DOF arm in our design. It gives some useful equations on how to move the arm in 3d space when we have individual control over all 3 individual joints.

Ramish, S. B. Hussain and F. Kanwal, "Design of a 3 DoF robotic arm," 2016 Sixth International Conference on Innovative Computing Technology (INTECH), Dublin, Ireland, 2016, pp. 145-149, doi: 10.1109/INTECH.2016.7845007.

General purpose hands for bin-picking robots

This paper describes how an advanced grabber design can be produced for the task of picking up objects from an unorganised bin.

The 2 main grabber methods of the paper are the clamping and the attracting methods.

First the paper describes the most important design requirements.

Then they describe 3 designs which accomplish the requirements:

1. The surface-adapting vacuum gripper (SAVG) uses a suction cup on a semi-rigid spring which can use a vacuum to stick to a flat surface of an object. This design also features contact sensing and a locking mechanism.
2. The contour-adapting vacuum gripper (CAVG) worked similarly to the SAVG but uses 20 smaller suction cups in a rectangle shape to have more grip on the object.
3. The parallel-jaw hand was a traditional gripper design with 2 parallel 'fingers' and which can move towards each other to grab an object from the sides.

Overall of these design the first SAVG will be sufficient for our project.

The cards in our project will always be picked up in the horizontal position so the parallel-jaw hand would not work well and the CAVG would be overkill since we do not need much force to lift the light cards.

Tella, R., Birk, J. R., & Kelley, R. B. (1982). General purpose hands for bin-picking robots. IEEE Transactions on Systems, Man, and Cybernetics, 12(6), 828-837.

Picking up operation of thin objects by robot arm with two-fingered parallel soft gripper

This paper gives a detailed sequence of events and mathematical force analysis to pick up paper or a card using only soft grippers.

In this paper they used a multi-DOF robot arm with 2 soft 'fingers', one with a hard and pointy 'nail' which can be used to slide under the paper.

The success rate for picking up a paper sheet is 90% but unfortunately only 10% when picking up a hard plastic card.

Given the very low success rate on rigid cards which we will use and the relatively costly components, we will probably not use soft grippers in our project so in hindsight this paper will not be very useful.

Yoshimi, T., Iwata, N., Mizukawa, M., & Ando, Y. (2012, May). Picking up operation of thin objects by robot arm with two-fingered parallel soft gripper. In 2012 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO) (pp. 7-12). IEEE.

Members

• Abel Brasse (1509128) - a.m.brasse@student.tue.nl
• Linda Geraets (1565834) - l.j.m.geraets@student.tue.nl
• Sander van der Leek (1564226) - s.j.m.v.d.leek@student.tue.nl
• Tom van Liempd (1544098) - t.g.c.v.liempd@student.tue.nl
• Luke van Dongen (1535242) - l.h.m.v.dongen@student.tue.nl
• Tom van Eemeren (1755595) - t.v.eemeren@student.tue.nl

Logbook

Week 1	Description of work done	Total time
Abel Brasse	Meeting time (3h). Literature search, searching, reading, summarizing papers (5h), User Study (2h)	10
Linda Geraets	Meeting monday (2h), Meeting wednesday (1h), Researching and reading papers (3h 30min), Summarizing papers (3h 30min), User Study (2h)	12h
Sander van der Leek	Monday & Wednesday meetings (3h), Literature search (2h), Reading papers (6h), Summarizing papers and drawing conclusions (2h)	13h
Tom van Liempd	Course Introduction (1h), Meetings (2h), Problem statement (1h 30m), Researching (4h), Reading and summarizing papers (3h)	11.5h
Luke van Dongen	Course Introduction (1h), Meetings (2h), Literature search (1h 30m), Reading Literature (6h), Summarizing literature (3h)	13.5h
Tom van Eemeren	Course introduction(1h), Brainstorming (1h), Prepared agenda (30min), group meeting (1h), literature search (1h), reading literature (6h), summarising literature(1h), added deliverables, approach, milestones and task division to the wiki (2h).	13.5h

Links

User interview