PRE2016 3 Groep16

Group members

Student ID	Name
0943957	M.D Visser
09	G. Marzano
09	R. Schalk
09	T. Jansen
09	J. Van Galen
09	B.G.M Hopman

Introduction

This is the Wiki page for the project Robots everywhere (0LAUK0) of group 16. The subject chosen is "Detection of people by using drones equipped with IR/heat sensors and Camera". Developing such a technology would make an impact due to the several applications it carries along. In the first phase of the project, after brainstorming, the following applications emerged:

• Find refugees in open sea
• Local security and criminals detection
• Inspection of insulation materials in houses (not directly "human recognition", but considered a valuable research)

Problem Statement

Due to turbulent geopolitical times in parts of Africa, thousands of people try to get a safe, better life in Europe. Conflicts in countries such as Somalia and the violation of human rights in countries with strict regimes, such as Eritrea, forces certain groups in those countries to move elsewhere. The safest place that is reachable for them is Europe. However, to make it to Europe, the Mediterranean Sea needs to be crossed. The crossing is often done with old, small boats that were originally intended for much less passengers than they are currently loaded with. The result is the sinking of many boats, resulting in many drownings. In 2016 there were 4,218 known deaths in the Mediterranean with drowning as cause. In the first month of 2017, the death count as cause of drowning was already 377 migrants.

Up until November 2014, Italy had its own rescue operation called Mare Nostrum to find refugees that were victims of boat accidents. The rescue mission was successful, about 160,000 people were saved from drowning. However, the use of seven ships, two helicopters, three planes and the help of the marine, coastal guard and Red Cross costed €9,5 million per month which was too expensive and the mission stopped. Moreover, it was thought that the rescue operation would encourage refugees to take their chances to cross the sea, as they would be rescued anyway when they would get in trouble. The result of stopping with the operation resulted in a ten fold of casualties.

The search for refugees is currently done with the use of helicopters and planes equipped with cameras. However, these aerial vehicles are very expensive to keep in the air. The cost of the assistance of a C-130 turboprop plane used in rescue missions, for example, costs more than €6,000 per hour. The main purpose for these aircrafts is to spot refugees, after which a boat will get them out of the water.

In order to make rescue operations such as Mare Nostrum stay in action, costs have to be cut. Since the aircraft division of those operations are one of the most expensive aspects, another solution must be invented. The usage of the promising drone technology could be a very good solution since they are a lot cheaper to keep flying. Moreover, they could be used in large numbers to cover large water surfaces making rescue operations much more efficient. Therefore, this wiki will discuss the feasibility of the deployment of drones within refugee rescue missions. Since drones are currently relying on rather weak power sources, batteries in particular, the equipment of the drones should be minimal. The use of a thermal camera will therefore be investigated because apart from the fact that it is small and lightweight, the automatic detection of organisms, in this case human refugees, can be easily implemented since humans have a warmer body temperature than the surrounding water. Moreover, since the thermal cameras are not dependent on visible light, the cameras can also be used at night increasing operational hours of the drone.

Objectives

General

• Showing that drones can be implemented in substitution of regular surveillance vehicles/cameras, consequently reducing costs and risks for operators.(Principle of Unnecessary Risk-PUR )
• Autonomous movement (User)

Application 1: Refugees search

• Reducing costs for refugee search in open sea (Society)
• Increase relative number of rescued refugees (Society)

Application 2: Surveillance & Security Drones

• Reduce the number of crimes (Society)
• Have a better understanding of crime distribution (Society)
• Make surveillance cheaper and more efficient (Enterprise & Society)
• Large scale surveillance

Application 3: Insulation in houses

• Verify heat losses in buildings (Society, Enterprise)
• Increased efficiency of procedure (e.g. analyze multiple living units & buildings at same time) (Enterprise)
• Reduction of costs for maintenance and related time (Enterprise)

Approach

General

• Determine the demands and benefits for user, society & enterprise.
• Divide group in subgroups working on different aspects of the project.
• Make a detailed planning.

Technical

• Equip the drone with a thermal camera (considering that a drone can be provided by TU/e).
• Program the drone and tune the sensors (e.g. find the threshold voltages) to detect the different values.
• Link the obtained data to an environmental structure (e.g. environment heat model).
• Adjust program in order to map properly the perimeters and consider the external heat deviations.
• Act on the environment under inspection (e.g. transmitting signal to operator).

Literature Study

For the literature study of this project the article titled ‘feasibility study of inexpensive thermal sensors and small UAS deployment for living human detection in rescue missions application scenarios’ was found. This article mentions how there are two critical phases in which geospatial imaging for rescuing purposes can be very useful. Namely for the detection of humans and secondly for the confirmation whether a detected human is dead or alive. Moreover the article elaborates on the “proof of concept for using small UAVs equipped with infrared and visible diapason sensors for detection of living humans in outdoor settings”. In which “Electro-optical imagery was used for the research in optimal human detection algorithms”.

Quite a lot of useful information about thermal imaging came forward in this research. Already in the introduction a human psychological aspect comes forward which says that “the human tendency to disregard opportunity costs when the life of identifiable individuals are visibly threatened. Due to this fact, we may observe operations when hundreds of people and multiple sets of equipment are deployed to save only one human life”. Moreover Rudol et al, already introduced human body detection via positioning algorithms using visible and infrared imagery in 2008. Follow up research in this field realized analysis algorithms that detect breathing and heartbeat rates through 15 cm of rubble.

In our research we were already aware of the fact that manned aerial vehicles for rescuing purposes are quite expensive. However something that came forward in this article and was not considered by us is that for manned aerial vehicles “very strict requirements are needed for areas of take-off and landing, and these areas are often far from the search and rescue area”. Moreover we found out that there exist several classes of UAVs, however we will adapt to the drone that can be provided by TU/e.

The research article conducted their research on human dummy objects and real humans. Experimental results showed that “various types of boundaries created by changes in feature signs such as color and texture, bringing a lot of difficulties in automated image processing. Thus, a potentially reliable algorithm needs to consider all combination of different types of image attributes together in order to provide correct segmentation of real natural images”. The conclusion following up on this result made clear that living humans can be detected in a reliable way in positive (13 °C) as well as negative (-5 °C) temperature surroundings.

Planning

To organize the workload of the project a planning is required, where the main requirements and deadlines are set, together with the task division (starting from week 3 of the quartile e.g. 20th February 2017).

Gantt chart for workload division during the project.

Logbook

Weekly the progress of the group is going to be reported in this logbook. The purpose is to keep the full group up to date with the progress of each sub group. Moreover, it is useful to be always able to compare the actual progress with the planning.

Week 1

During week 1 the group has been formed, the possible objectives of the project have been discussed and for each of the possibility the main USE aspect have been identified, as reported in Objectives section. Moreover, a rough estimation of the possible planning has been discussed, to find out the best suitable times for the group members to meet.

Week 2

During week 2 the group had to present the subject of the project, with the relative objectives and approach. Based on the feedback received, the group starts to focus on one of the application presented: The Refugees Search.

USE analysis of the application

User

The users of the rescue drone will be the coastal guards Italy, Greece, etc. Since deploying ships and planes for surveillance is costly and slow, the use of drones can decrease costs significantly and increase search speed. Due to the reduction in costs, a lot of the drones can be deployed, increasing the surveillance area. Deployment of drones is also easier for the operator, since you can fly drones in autonomous mode. This makes it possible to control the drone by anyone, instead of requiring skilled pilots. Automatic detection of refugees by the drone, simplifies locating them, after which rescue ships can be send to their location. The user requirements, are that the drone should be easily deployable and be able to detect refugees with as little control by the operators as possible.

Society

Rescue drones will benefit society, since a lot of deaths of refugees can be prevented by their deployment. A reduction in costs, will make more money available for the shelter and care of the refugees. Better shelter and care will reduce tensions, decreasing civil unrest. This way the refugee crisis will possibly have a more positive outcome. Requirements for society are to reduce the costs of rescue operations, saving money.

Enterprise

A rescue drone has little commercial applications, but a drone equipped with thermal camera, could be utilized by insulation companies to scan houses in urban areas for improvements. It could also be used by security companies, to scan business premises instead of fixed cameras that have a limited view.

Demonstration

To demonstrate the functionality of the drone, the detection of a human being in a marine environment will e inspected. This could be shown by a test in a swimming pool or in the 'Dommel', with an individual swimming in the water. If the drone is able to recognize whether there is a human in the water or not, it will be a passed test. The test will be recorded on video and shown at the final presentation of the project.

State of the art

(Boat) refugee drone Avy

The use of drones is a development that has not been around for decades. This directly implies that research on drones for application in specific domains is not largely spread and therefore there are few state-of-the-art systems that are comparable with the idea that this group wants to develop. Nevertheless, there has been developed a drone that is specifically designed for refugees.

Avy is the drone that has been receiving publicity lately, as it focusses on delivering goods (a huge floater) to the refugees on open see. After dropping the floater, the location of the refugees is directly known and the emergency services can move to the area with the gathered knowledge. The weakness of this design is that the drone does not focus on the detection of refugees, as the location that is transmitted comes down to the location at which the floater is dropped. For dropping a floater, one must be able to detect the refugees by some means, as dropping a floater randomly will not help the current problem. A thermal camera, with which our design will be equipped, will ensure detection of refugees on open see, even at night conditions. Many other elements of Ivy are very hopeful and inspirational to our development, such as the design of the Avy.

The design of Avy focused on traveling large distances overseas, which is not doable with a 'regular' quadcopter drone. This, however, is not directly interesting for our project, as the main focus is on the detection by using the thermal camera instead of designing a drone that is able to cover large distances. Even though the design of Avy focussed on large distance travelling on high speeds (around 200 km/h), it is still able to take-off and land vertically. Moreover, the design is fully electric driven with propulsion, able to fly (nearly) autonomously and able to carry a payload of around 10 kg.

"UAE Drones for Good Award", Dubai

In Dubai, since 2016 an annual competition called "UAE Drones for Good Award" has been established. The mission, as stated on their website follows:

The UAE Government invite the most innovative and creative minds to find solutions that will improve people’s lives and provide positive technological solutions to modern day issues.

The Avy case, above described, has been one of the finalists of this competition along with numerous other intriguing projects. Some of them were focused on rescuing and assisting in case of disasters, namely "Drones for Search and Rescue" and "FINDER drone". One of the finalist projects, called "Drones 4Right2Life", is particularly interesting because of the similarities it has with the objectives of this group.

On the page of the project , they affirm:

The solution promises to have [...] an earlier, closer, and more efficient detection of the vessels, which may result not only in a better use of human and technological resources in the rescue effort, but more importantly, in saving more lives.

Week 3

In preparation for the presentation of this week, the group defines the deliverables of the project

Deliverables

• A working drone equipped with thermal-sensing camera capable of detecting humans
• A demonstration video which shows the functionalities of the drone
• Complete wiki with all relevant information on the project. This includes:
  • introduction
  • context analysis (state of the art)
  • problem statement
  • research with an accent on the USE perspective
  • conclusion with some ideas for further improvement.

Week 4

During the carnival break, the group took contact with dr. ir. M.J.G. van de Molengraft and subsequently received the drone. Moreover, the literature study has been completed and improved. An important discussion took place during Week 4, regarding the use of the thermal camera for recognition. The reasons which lead to the propose of such a technology can be found in the previous section of the wiki. However, an alternative might be to use a regular camera and, instead of showing the actual working principle, just demonstrate it by mean of a bright colored object. Using colors that can be easily distinguished from the water in normal situations, eliminates the need for a thermal camera. This and other possible options have been discussed during the weekly meeting with the professors. The feedback received has been used to start on working on a list of requirements.

Work on the Drone

After having received the drone, the group started to work on it. In order to make it more convenient to control the drone, it was decided to use matlab to send commands and receive data. The idea was that within matlab we could also do the image analysis. However, after spending hours testing different matlab codes, still no success was accomplished since the drone would not get into the air, nor matlab was able to receive data from the drone.. After some time we figured that one of the codes, received from the TU/e, is only suitable for a previous version of the drone. As said, different other codes have been tested, but they also do not seem to get the drone into the air.

Week 5

At this point, some adjustment had to be made to the original planning, due to some changes of direction of the project. These mainly concern the type of camera used, which is going to be a regular camera instead of a thermal one. Consequently, to clarify more which objectives have to be achieved, the final requirements have been listed. Moreover, during this week and the end of the previous one, some work has been done on the image processing as well. This is necessary for the detection of refugees through the camera. Finally, the requirements are also important for preparing the final demonstration video, and assess how valuable the outcome is going to be even though a different camera is used.

List of Requirements

A list of requirements has been done, in order to proceed efficiently in the following weeks:

• Drone
  • Minimum height: 10m
  • Minimum speed: 1 m/s
  • Minimum angle of detection (with respect to the perpendicular) : 10 °
  • Minimum flying autonomy : 2 minutes
  • Minimum detection rate : 70%

• Camera
  • Minimum resolution: Qvga (bottom camera & front camera)
  • Minimum

Considering the supplied drone and the equipped cameras, the requirements are met. Furthermore, these requirements are going to be used to set up the final demonstration, where the functionality of the final design will be shown.

Image Processing

The group started to experiment the image processing using MATLAB. JASPER….