Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3

From Control Systems Technology Group

(Difference between revisions)
Jump to: navigation, search
(Internal Validation)
Line 415: Line 415:
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met.  
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met.  
-
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to fill in certain propositions which required this knowledge. Examples are the size of the airport and the amount of daily departures and arrivals. After filling the questionnaire, we filled the results of this questionnaire into our decision model, and looked at the outcomes. Together, we discussed whether these results do indeed make sense.  
+
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into accoutn were among other the size of the airport and the amount of daily departures and arrivals.  
 +
 
 +
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation:
 +
 
 +
1.
 +
2.
 +
3.
 +
4.
 +
5.
 +
6.
 +
7.
 +
8.
 +
9.
 +
10.
 +
 
 +
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:
 +
 
 +
* insert picture of result *
 +
 
 +
Now, we discussed with all the group members whether these outcomes did make sense.  
----
----
Back to the [[PRE2018_3_Group4 | root page]].
Back to the [[PRE2018_3_Group4 | root page]].

Revision as of 12:47, 3 April 2019

Contents

Decision Model Validation

Introduction

When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used[1].

What now?

Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system[1]. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.

Credibility

As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?

As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport.

Methods

Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.

Evaluation

Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.

The questionnaire we propose can be observed below.

Questionnaire

This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.

We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.

For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.

This questionnaire also has a PDF-format, which can be found here.

General questions

We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved.

  • What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?
  • What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?
  • How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.

Detection

1. I want to be advised on an anti-UAV detection solution

  • Agree
  • Neutral
  • Disagree

Category: Need for a solution

Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.

Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.

2. The detection system must be able to detect UAVs within a range of 4000 meters

  • Agree
  • Neutral
  • Disagree

Category: Range

Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.

Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.

3. The detection system must detect illegal UAV presence within less than 1 second

  • Agree
  • Neutral
  • Disagree

Category: Speed of Operation

Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.

Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.

4. The detection system must not make any loud noises annoying people around the airport

  • Agree
  • Neutral
  • Disagree

Category: Disturbance of the environment

Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.

Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.

5. The detections system must be able to detect UAVs from all the categories(C1-C4)

  • Agree
  • Neutral
  • Disagree

Category: Effect on Different Types of UAVs

Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.

Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.

6. The detection system must be able to scale with the growth of the airport in size

  • Agree
  • Neutral
  • Disagree

Category: Scalability

Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.

Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.

7. The detection system must be able to detect multiple UAVs concurrently

  • Agree
  • Neutral
  • Disagree

Category: Number of Drones it Can Handle

Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.

Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.

8. The detection system must not emit any CO2

  • Agree
  • Neutral
  • Disagree

Category: Emission

Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.

Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.

9. The detection system must not be larger than 1 m3

  • Agree
  • Neutral
  • Disagree

Category: Size

Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.

Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.

10. The detection system must be able to identify the UAV properly

  • Agree
  • Neutral
  • Disagree

Category: Identification

Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.

Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.

11. The detection system must be able to detect UAVs automatically without needing any human interaction

  • Agree
  • Neutral
  • Disagree

Category: Level of Autonomy

Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction.

Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.

12. The detection system must be able to operate in the event of a power outage

  • Agree
  • Neutral
  • Disagree

Category: Power Outage

Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.

Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.

13. The detection system must be able to operate under any weather condition

  • Agree
  • Neutral
  • Disagree

Category: Weather

Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.

Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.

14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)

  • Agree
  • Neutral
  • Disagree

Category: Time

Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.

Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.

15. The detection system must be able to detect UAVs at night

  • Agree
  • Neutral
  • Disagree

Category: Time

Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.

Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.

16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation

  • Agree
  • Neutral
  • Disagree

Category: Portability

Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.

Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.

Neutralisation

1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised

  • Agree
  • Neutral
  • Disagree

Category: Danger to Humans

Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most likely fall to the ground. Other solutions, however, do not have this issue.

Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.

4. The neutralisation system must not emit any CO2

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

5. The neutralisation system must be suitable to use in locations close to residential areas

  • Agree
  • Neutral
  • Disagree

Category: Disturbance to the Environment

Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.

Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.

6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

7. The neutralisation system must be able to neutralise commercial UAVs

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

8. The neutralisation system must be easy to extend

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

10. The neutralisation system must be able to neutralise UAVs under any weather circumstance

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

11. The neutralisation system must be able to operate 24/7

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

12. The neutralisation system must be able to neutralise UAVs at night

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

14. The neutralisation system must be able to be used without training of the employees

  • Agree
  • Neutral
  • Disagree

Category: Level of Training

Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some other solutions are much easier to use.

Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.

15. The neutralisation system must be able to operate in the event of a power outage

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

16. The neutralisation system must be able to neutralise UAVs without human input

  • Agree
  • Neutral
  • Disagree

This proposition has been explained and motivated in the section for detection.

Closing questions

It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.

  • What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?
  • What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?
  • Other remarks

Thank you for filling in this questionnaire.

Validation by Domain Experts

As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times.

Internal Validation

So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met.

So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our airport analysis to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into accoutn were among other the size of the airport and the amount of daily departures and arrivals.

So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation:

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:

  • insert picture of result *

Now, we discussed with all the group members whether these outcomes did make sense.


Back to the root page.

References

  1. 1.0 1.1 Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf
Personal tools