Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3

From Control Systems Technology Group
Jump to navigation Jump to search

Decision Model validation

Introduction

When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used[1].

What now?

Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system[1]. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.

Credibility

As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?

As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport.

Methods

Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.

Evaluation

Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.

The questionnaire we propose can be observed below.

Questionnaire

This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.

We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.

For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.

This questionnaire also has a PDF-format, which can be found here.

General questions

We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved.

  • What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?
  • What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?
  • How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.

Detection

1. I want to be advised on an anti-UAV detection solution

  • Agree
  • Neutral
  • Disagree

Category: Need for a solution

Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.

Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.

2. The detection system must be able to detect UAVs within a range of 4000 meters

  • Agree
  • Neutral
  • Disagree

Category: Range

Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.

Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.

3. The detection system must detect illegal UAV presence within less than 1 second

  • Agree
  • Neutral
  • Disagree

Category: Speed of Operation

Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.

Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.

4. The detection system must not make any loud noises annoying people around the airport

  • Agree
  • Neutral
  • Disagree

Category: Disturbance of the environment

Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.

Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.

5. The detections system must be able to detect UAVs from all the categories(C1-C4)

  • Agree
  • Neutral
  • Disagree

Category: Effect on Different Types of UAVs

Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.

Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.

6. The detection system must be able to scale with the growth of the airport in size

  • Agree
  • Neutral
  • Disagree

Category: Scalability

Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.

Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.

7. The detection system must be able to detect multiple UAVs concurrently

  • Agree
  • Neutral
  • Disagree

Category: Number of Drones it Can Handle

Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.

Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.

8. The detection system must not emit any CO2

  • Agree
  • Neutral
  • Disagree

Category: Emission

Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.

Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.

9. The detection system must not be larger than 1 m3

  • Agree
  • Neutral
  • Disagree

Category: Size

Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.

Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.

10. The detection system must be able to identify the UAV properly

  • Agree
  • Neutral
  • Disagree

Category: Identification

Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.

Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.

11. The detection system must be able to detect UAVs automatically without needing any human interaction

  • Agree
  • Neutral
  • Disagree

Category: Level of Autonomy

Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction.

Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.

12. The detection system must be able to operate in the event of a power outage

  • Agree
  • Neutral
  • Disagree

Category: Power Outage

Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.

Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.

13. The detection system must be able to operate under any weather condition

  • Agree
  • Neutral
  • Disagree

Category: Weather

Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.

Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.

14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)

  • Agree
  • Neutral
  • Disagree

Category: Time

Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.

Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.

15. The detection system must be able to detect UAVs at night

  • Agree
  • Neutral
  • Disagree

Category: Time

Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.

Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.

16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation

  • Agree
  • Neutral
  • Disagree

Category: Portability

Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.

Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.

Neutralisation

1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised

  • Agree
  • Neutral
  • Disagree

Category: Danger to Humans

Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most likely fall to the ground. Other solutions, however, do not have this issue.

Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.

4. The neutralisation system must not emit any CO2

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

5. The neutralisation system must be suitable to use in locations close to residential areas

  • Agree
  • Neutral
  • Disagree

Category: Disturbance to the Environment

Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.

Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.

6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

7. The neutralisation system must be able to neutralise commercial UAVs

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

8. The neutralisation system must be easy to extend

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

10. The neutralisation system must be able to neutralise UAVs under any weather circumstance

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

11. The neutralisation system must be able to operate 24/7

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

12. The neutralisation system must be able to neutralise UAVs at night

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

14. The neutralisation system must be able to be used without training of the employees

  • Agree
  • Neutral
  • Disagree

Category: Level of Training

Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some other solutions are much easier to use.

Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.

15. The neutralisation system must be able to operate in the event of a power outage

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

16. The neutralisation system must be able to neutralise UAVs without human input

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

Closing questions

It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.

  • What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?
  • What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?
  • Other remarks

Thank you for filling in this questionnaire.

Validation by Domain Experts

As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times.

Internal Validation

So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met.

Picking an airport to use for validation

So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behalf. We used the information from our airport analysis to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.

Filled in proposition on behalf of this airport

So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation:

Detection

1. Q: I want to be advised on an anti-UAV detection solution.

A: Agree

Motivation: Because we as an airport are looking for a system that is able to detect UAVs.


2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.

A: Disagree.

Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.


3. Q: The detection system must detect illegal UAV presence within 1 second.

A: Agree.

Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.


4. Q: The detection system must not make any loud noises annoying people around the airport.

A: Agree.

Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution.


5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.

A: Agree.

Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless.


6. Q: The detection system must be able to scale with the growth of the airport in size.

A: Agree.

Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box mandatory property.


7. Q: The detection system must be able to detect multiple UAVs concurrently.

A: Disagree.

Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution.


8. Q: The detection system must not emit any CO2.

A: Agree.

Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction.


9. Q: The detection system must fit in an area of 0.5m^3.

A: Neutral.

Motivation: We have plenty of space, so we do not really care about the size of the solution.


10. Q: The detection system must be able to properly identify the UAV.

A: Disagree.

Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us.


11. Q: The detection system must be able to detect UAVs automatically.

A: Agree.

Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.


12: Q: The detection system must be able to operate in the event of a power outage

A: Disagree.

Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals.


13: Q: The detection system must be able to operate under any weather condition.

A: Agree.

Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible.


14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).

A: Agree.

Motivation: We may have long days at the airport, and then we want the detection system to be working all the time.


15: Q: The detection system must be able to detect UAVs at night.

A: Agree.

Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones.


16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.

A: Disagree.

Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.

Neutralization

17. Q: I want to be advised on a anti-UAV neutralisation solution.

A: Agree.

Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.


18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.

A: Agree.

Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.


19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.

A: Disagree.

Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.


20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.

A: Agree.

Motivation: We do not want people to get hurt by our anti-UAV solution.


21. Q: The detection system must not emit any CO2.

A: Agree.

Motivation: The same as for the detection system.


22. Q: The neutralisation system must be suitable to use in locations close to residential areas.

A: Disagree.

Motivation: Our airport is not located near residential areas, so we do not have to worry about that.


23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to neutralise UAVs (possibly privately built) that use other communication methods.

A: Agree.

Motivation: The same as for the detection system.


24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.

A: Agree.

Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.


25. Q: The neutralisation system must be easy to extend.

A: Agree.

Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.


26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.

A: Disagree.

Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.


27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.

A: Agree.

Motivation: The same as why the detection system must be able to operate under any weather circumstances.


28. Q: The neutralisation system must be able to operate 24/7.

A: Agree.

Motivation: The same as why the detection system must be able to operate 24/7.


29. Q: The neutralisation system must be able to neutralise UAVs at night.

A: Agree.

Motivation: The same as why the detection system must be able to detect UAVs at night.


30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.

A: Agree.

Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.


31. Q: The neutralisation system must be able to be used without training of the employees.

A: Disagree.

Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees.


32. Q: The neutralisation system must be able to operate in the event of a power outage.

A: Disagree.

Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.


33. Q: The neutralisation system must be able to neutralize drones without human input.

A: Disagree.

Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.

More important propositions

There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:

Proposition 3: he detection system must detect illegal UAV presence within 1 second.

Motivation: We find it very important for the anti-UAV system to be quickly deployable.


Proposition 11: The detection system must be able to detect UAVs automatically.

Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.


Proposition 13: The detection system must be able to operate under any weather condition.

Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time.


Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.

Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness.


Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.

Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.


Proposition 25: The neutralisation system must be easy to extend.

Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.


Proposition 27: The neutralisation system must be able to neutralize drones under any weather circumstance.

Motivation: See motivation of proposition 13.

Outcome of the decision model for this airport

After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:

Missing image
Figure 1: Picture of the best solutions for this airport from the decision model.

As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.

If any reader would be interested in reproducing these results, one could go to drones.jortdebokx.nl (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.

Discussion of the results for this airport

Now, we discussed with all the group members whether these outcomes did make sense. It was interesting to see that there are no solutions that have a higher matching percentage than 77% with our mock-up airport. However, after looking through all the solutions that we have gathered, we indeed concluded that these top 7 solutions would be the best fit for our airport. So apparently there is no solution in the list that almost perfectly matches the expectations of the mock-up airport. After close investigation of the highest matched solution attributes against the needs of our mock-up airport, we indeed see that the top solutions mostly cover the airport's needs for a large part. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications. We could also cover all other propositions and compare these with the attributes of the solutions but that would simply not be very interesting and also would only just validate the logic behind the code of the web app.

What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. This is also not a solution that any airport would not have come up with themselves. We did, however, include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. As well as giving the users a comparison between such simple solutions and more sophisticated solutions, such that the users can decide for themselves whether these differences are worth investigating in. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a lower match rating. After performing our internal validation, we indeed found this to be the case. It was unfortunate that our decision model could not be validated by external users, however, we hope that this validation, whilst far from perfect, gives some impression of the performance/accuracy of the decision model.

Conclusion of the validation for this airport

We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning.

Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test.


Back to the root page.

References

  1. 1.0 1.1 Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf