AutoRef implementation

The implementation for the AutoRef autonomous referee for RoboCup Middle Size League (MSL) robot soccer is the proposed design of the AutoRef system.

In 2021 contributions by MSD 2020 focused on detecting ball-to-player distance violations.

Introduction

Objective statement

The main objective of the implementation part of the project was to detect violations of the rules related to the distance between the ball and players during the following game states:

• Free kick
• Kick-off
• Corner kick
• Goal kick
• Throw-in
• Dropped-ball
• Penalty kick

These rules are described in Laws number 8, 10, 13, 14, and 15 of the MSL Rulebook.

Motivation

This objective was chosen due to several reasons:

- Past projects analysis has shown that this functionality has never been designed before

- Stakeholder interviews (the MSL referees) have led to the conclusion that this kind of rules are hard to control for a human being

- Proof of concept for the developed functional specification was desirable

- Learning goals of the team members correspond to the technical solutions necessary for the functionality development

Scope of work

The following topics were included in the implementation scope:

• Requirements formulation

• Architectural decomposition development

• Individual code blocks development

• Individual code integration

• Software testing on images and videos

These topics are explained in detail in the following sections.

Process model

Introduction

The development activities of the design team need support from process models. In this project, the V-model is chosen to guide the development procedure from requirement engineering to system validation. Due to the particularity of the project itself, some details of the model have been changed. At the same time, the agile approach was used during system development and combined with the traditional V-model, which makes the project progress more flexible and efficient.

Use of V-model

V-model has the following advantages for the development of the project:

- Design team's project is based on machine vision and software algorithms. V-model was first proposed in the software development environment and has matured in the software development field.

- The project team has five members, all of whom can participate in the development of subsystems and they can be developed at the same time. On the premise that the system architecture is determined, V-model can greatly improve the development efficiency.

- The system development starts from the fourth week, which means that the team needs to complete the system development in five weeks, and the mature and ready-made V-model process can save a lot of time spent on project management.

Left:V-model used for guiding the project process for design team; Right: Test plan derived from the V-model

Based on the general V-model, a detailed test plan has been made for the verification of the system both from functional and performance perspectives.

Based on the requirements derivation results, functional and performance requirements were set up and related tests were planned as shown in the figure above. In this plan, the details of the V-model are supplemented, and more detailed test steps and iterations are added in the test and verification phases. The technical blocks were integrated into the first phase, then several images regarding typical use cases are created from the simulation environment (refer to Section 3) in order to verify the functional requirements. Videos were created to test the performance of the system based on particular scenarios (refer to Section 4). Code was updated iteratively after several times of tests. After the code was verified, a simulated game video was created in the simulation environment to illustrate how the system works in the 'real' world.

Agile approach

Due to the limited project time and various uncertainties in the system development process, Agile approach was applied in the system development process, which is mainly reflected in the system architecture and design choice part.

There are two main difficulties in the development of this project:

- How to implement an efficient and fast detection algorithm?

- How to achieve an accurate image capture in reality?

Usually, the algorithm needs to be executed after the system obtains the image, but it is worth noting that the design of vision system has the following two considerations, which greatly increases the complexity of the system design:

- Fixed camera OR moved camera

- One camera OR multiple cameras

After careful evaluation, we thought that time spent on algorithm development will be greatly reduced in the design of the vision system, which is not the result we want. Therefore, we finally determined the system development scheme based on Agile. That is, a single fixed camera is initially used, and a simulated game situation has been created under the software simulation environment as a reference sample for algorithm development. Optimization of vision system can be carried out after the algorithm is developed.

The main idea is quickly designing and checking the performance of the algorithm we developed, which also confirms the rationality of ‘decision fast’ in the Agile approach.

Major design choices

Programming language

MATLAB was chosen as the programming language due to the availability of in-built functions and documentation, which is useful in a quick test of a proof-of-concept of the implemented algorithm.

Selection of test environment

It was decided to use a simulation environment, to quickly test and verify the functionality of the ball-player distance violation check algorithm. It was desirable to verify the implementation in a simulation environment before committing to a specific choice of hardware. Other factors leading to the choice of a simulation environment included—time limitations, limited access to the tech-united playing field (covid restrictions), and the limited availability of match video footage with the desirable qualities (RGB top-view).

A custom simulation environment was built using MATLAB, Simulink 3D animation, and the Virtual Reality Modelling Language (VRML). Another option was to use GreenField, the visualization software used by Tech United in replaying recorded match data. This was not done due to less familiarity with Linux systems in the team, and a potential dependency on Tech-united developers, which could lead to a time-consuming learning curve.

Vision system parameters

The main design choices to be made were regarding the selection of the vision system, involving the following aspects:

1. Mobility of the camera(s) (static vs moving)
2. Location(s) of the camera(s)
3. Camera resolution

Motivations for choosing a static camera were as following:

• Simplified localization requirements
• Simplified implementation architecture
• Less risk of invasiveness (spatial and audio)

Motivations for selecting a top camera view were as following:

• Using a top-view camera, projection errors can be minimized, which makes the implementation easier
• Having a movable top view camera that stays right above the ball would help avoid perspective distortions to a greater extent, but:

• We would need to account for localization uncertainty in the case of drones
• Multiple cameras might still be needed to detect events that are not in the vicinity of the ball

• It would be possible to view the entire field using a camera of sufficient height or field of view

Parameters to consider when using a top-view camera:

• Mounting height
• Field of View
• Resolution or Image size

Parameters to consider for the ball-player distance violation decision-making algorithm:

• Pixel to meter ratio or Resolution
• Frequency rate—The frequency rate required for the decision-making algorithm is defined based on

• The pixel-meter ratio,
• Worst-case scenarios defined considering robot dimensions, speed, and acceleration limits

• Maximum allowed perspective distortion

• When using a single top camera, perspective distortions are unavoidable
• High distortions can affect the visibility of the ball, separation of team players, and also lead to incorrect position estimate of the players.

Possible issues of using a single static top camera

• Perspective distortions

• Objects that are not directly below the camera would be seen at an oblique angle

• Occlusions

• The ball may not be completely visible and be occluded by players
• Players that are too close to each other may not be detected separately using a simple camera

• Limitations on accuracy

• Placing the camera very high above the field, in such a way that the entire field is visible, could lead to objects being very small in the images, and affect the accuracy of detections
• Going with a higher resolution camera would improve the accuracy of the field

Final decision:

Considering time limitations, and the implementation complexity related to the use of multiple cameras and moving cameras, a decision was made to test the ball-player distance violation algorithm for a single top camera concept. The camera was considered to be at a height of 12 m above the center of the field and facing downwards. A choice was also made to keep the field of view to a manageable value of 1.2 radians, such that perspective distortions are minimized. Meanwhile, to have a manageable resolution, the images taken were considered to be Full HD (1920 x 1080) RGB images.

Requirements

Functional requirements

Formal formulation:

The first and foremost sub-requirement is that the system must be modular. Other sub-requirements are as follows:

• The system must detect the players and the ball inside the soccer field boundaries and identify the players’ team.
• The system must detect the different zones inside the soccer field (corner area, penalty area, etc.)

And the main requirements are:

• The system must check if the distance between the center of the ball and any part of the attacker team members (except for the kicker) before free kick, corner kick, kick-off, goal kick, and throw-in is not less than 2m. (with acceptable 5cm inaccuracy). One of the robots may stay anywhere inside the penalty area (except for the goal area) of its own team, even if the distance to the ball is shorter than 2m.

• The system must check if the distance between the center of the ball and any part of the defender team members before free kick, corner kick, kick-off, goal kick, and throw-in is not less than 3m. (with acceptable 5cm inaccuracy). One of the defender robots may stay anywhere inside the penalty area (except for the goal area) of its own team, even if the distance to the ball is shorter than 3m.

• The system must check if the distance between the center of the ball and any part of all the players before the dropped-ball is 1m. (with acceptable 5cm inaccuracy in this distance.). One of the robots may stay anywhere inside the penalty area (except for the goal area) of its own team, even if the distance to the ball is shorter than 1m.

Why the requirements are formulated with these numbers?

The distance value between the center of the ball and the players before certain events has been specifically mentioned in the law book.

The players' length and width are 50cm, 10% of the players' size was considered as the acceptable accuracy tolerance for the system.

How the requirements were formulated:

Ball-player zone violation is a task that enforces several laws. In order to come up with functional requirements for this task, the first step was to detect the required skills necessary for implementing this task.

The skills are as follows:

• ball detection (currently known as ball identify)
• player detection (updated: player identify)
• player team identification (updated: player team classify)
• zone detection (updated: field area classify)

The second step was to detect the related laws for each skill. In this order, the tasks that are part of that law could be identified. This step is an important one because when going into detail for skill implementation, it is important to keep in mind this skill is going to be used in those other tasks in the future and the implementation must be as general and adaptable for future use as possible. For keeping the system modular, it is necessary to specify requirements for the system in this way.

The identified laws are as follows:

• Law number 8 (kick-off)
• Law number 10 (the method of scoring)
• Law number 13 (free kick)
• Law number 14 (penalty kick)
• Law number 15 (throw-in kick)

For example, the identified tasks for these laws that are related to the ball detection skill are:

• Measure the ball’s velocity, for checking if it is stationary (from law number 13)
• Detect which player has taken the ball (form law number 10)
• Measure how much the ball has rolled before it has touched by other players (form law number 8)

For another example, the identified tasks for player detection skill are:

• Control players’ location
• Count the number of players
• Detect the kicking player
• Detect other players

And the process for other skills is the same.

The third step is to come up with the actual requirements and analyze them.

Performance requirements

Frequency

Formal formulation:

The system must be able to realize the functional requirements (based on the system accuracy) at least every 89 ms in order to avoid false-negative detections, which means it should have a detection frequency of 11.22Hz.

Architecture decomposition

The software architecture used for the implementation of the distance violation checking algorithm is shown in the figure below.

Architecture of the algorithm including inputs and code blocks

The algorithm is applicable during the following game situations—kick-off, free-kick, throw-in, goal-kick, corner-kick, drop-ball, and penalty-kick, when the ball has been placed at the desired location by the referee, and before the transition to the ‘ball-in-play’ situation. During these situations, players from the attacking and defending teams need to maintain minimum distance requirements from the ball. These distance requirements vary depending on the state of the game (e.g.: during kick-off, throw-in, free-kicks, corners and goal-kicks, the attacking team players other than the kicker should be at least 2 meters away from the ball center, while during an in-game penalty kick, all players other than the kicker should be 3 meters away from the ball center ) The scope of the implemented block could be more clearly visualized by referring to state diagram related to throw-in enforcement (REFER to DIAGRAM IN WIKI). The block is applicable in

1. the throw-in positioning state, when the ball has been placed at the required location, and
2. the throw-in start state when the players have positioned themselves correctly with respect to the thrower.

Inputs

1. Game state: the algorithm requires information on the state of the game, which determines the radii of the areas of interest.
2. Attacking team ID: The team which takes the kick is the attacking team, and the algorithm requires information on whether team A (left-side) or team B (right-side) takes the kick. In the case of dropped ball however, no ID is needed, as neither team gets to take the ball first.
3. A 2-D image identifying penalty and goal areas on the field, obtained from a calibration image (RGB, Full HD) of the empty field.
4. Ball color values (RGB) and tolerance on the values.
5. Player marker color values (RGB) and the tolerance on the values.
6. Current camera image (RGB, Full HD).

Sub-blocks

The sub-code blocks are briefly discussed below, further details are given in the subsequent sections

• Pre-processing blocks

• (zone of field function+ get penalty zones)—separates out the penalty and goal areas of the football field, given an input calibration image, and estimates the meters-pixels ratio.

• Internal blocks

• Blocks used only for the first iteration of ball-player-distance-violation check algorithm, given the reference location of the ball.

• Area of interest function

• Blocks used in each iteration of the ball-player distance violation check algorithm

• Ball detection algorithm
• Player detection algorithm
• Player classification algorithm
• Decision-making algorithm

Explanation of individual code blocks

Area of interest

Area of Interest (AOI) mentions the area that the violation may happen. This shows the difference between this scheme and the distance detection scheme. The reason of using AOI to detect the violating player rather than calculating the distances between each player and the ball is as follows:

According to the rulebook, none of the parts of a robot should be in certain distance with regard to the ball. Since robots may have various shapes, it is difficult to realize the detection of the closest point on a complex shaped boundary with regard to the ball.

Algorithm

1. Create a zero matrix with the same size as the image. First, an RGB image is input into the block and is binarized into a 2D matrix. Then the number of rows and columns are measured and used to create a new 2D matrix with zeros.
2. Specify different AOIs based on game states. Since the AOI radius are different based on different laws, the radius of two circles can be specified as the input of the block and the map-pixel ratio can be used to calculate the pixel radius for inserting circles in the following step.
3. Insert circles and create the output matrix. Firstly, a red filled circle is inserted in the first blank image while a green circle is inserted in the second blank image. Then an AOI matrix can be created with two layers by comparing the pixel value of red layer of first image and the pixel value of the green layer of the second image with 0.5 respectively (if truth, pixel value becomes 1). Finally, a 3D matrix with two layers can be created as AOI for the following processing. Both layers contain the labelled circles with 1.

Inputs

1. image_frame - 3 channel RGB matrix of the image frame of the game video
2. center_ball – The matrix coordinate of the center of the detected ball
3. rad_meter_1 – The radius of the inner circle based on the typical game state
4. rad_meter_2 - The radius of the outer circle based on the typical game state
5. MPratio – The ratio showing the number of pixels used to represent the distance in the real world.

Outputs

1. AOI – a 3D matrix showing the information of the inner circle and outer circle in separate layers.