https://cstwiki.wtb.tue.nl/api.php?action=feedcontributions&user=M.g.d.quincey%40student.tue.nl&feedformat=atomControl Systems Technology Group - User contributions [en]2024-03-28T18:29:12ZUser contributionsMediaWiki 1.39.5https://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71319Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:59:06Z<p>M.g.d.quincey@student.tue.nl: /* Conclusion of the validation for this airprot */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
=== Picking an airport to use for validation ===<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.<br />
<br />
=== Filled in proposition on behalf of this airport ===<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
==== Detection ====<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
==== Neutralization ====<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
==== More important propositions ====<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
=== Outcome of the decision model for this airport ===<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
=== Discussion of the results for this airport ===<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
=== Conclusion of the validation for this airport ===<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71318Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:58:57Z<p>M.g.d.quincey@student.tue.nl: /* Discussion of the results for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
=== Picking an airport to use for validation ===<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.<br />
<br />
=== Filled in proposition on behalf of this airport ===<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
==== Detection ====<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
==== Neutralization ====<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
==== More important propositions ====<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
=== Outcome of the decision model for this airport ===<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
=== Discussion of the results for this airport ===<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71317Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:58:50Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
=== Picking an airport to use for validation ===<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.<br />
<br />
=== Filled in proposition on behalf of this airport ===<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
==== Detection ====<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
==== Neutralization ====<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
==== More important propositions ====<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
=== Outcome of the decision model for this airport ===<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71316Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:58:43Z<p>M.g.d.quincey@student.tue.nl: /* More important propositions */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
=== Picking an airport to use for validation ===<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.<br />
<br />
=== Filled in proposition on behalf of this airport ===<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
==== Detection ====<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
==== Neutralization ====<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
==== More important propositions ====<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71315Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:58:36Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
=== Picking an airport to use for validation ===<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.<br />
<br />
=== Filled in proposition on behalf of this airport ===<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
==== Detection ====<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
==== Neutralization ====<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71314Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:58:25Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
=== Picking an airport to use for validation ===<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.<br />
<br />
=== Filled in proposition on behalf of this airport ===<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
==== Detection ====<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71313Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:58:12Z<p>M.g.d.quincey@student.tue.nl: /* Picking an airport to use for validation */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
=== Picking an airport to use for validation ===<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals.<br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71312Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:57:17Z<p>M.g.d.quincey@student.tue.nl: /* Discussion of the results for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
What is also good to see is that some solutions that we included, of which we knew would not be ideal solutions, were not included in the top matches. For example, we also included human sight as a detection system, or eagles as a neutralization system, but all these solutions had a very low match compared to the others. For example, human sight only had a match of 20.5%, which makes sense since this is not a very good solution in the first place. We did however include these solutions to give as much freedom for the users, and also to validate that solutions which are less good are not picked that often. After all, good solutions having a good match is important, but it is also important that solutions that are not so good also get a match that is not so good.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71310Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:53:30Z<p>M.g.d.quincey@student.tue.nl: /* Discussion of the results for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. After looking through all the solutions that we have gathered, we indeed concluded that these solutions would be the best fit for our airport. First and foremost were the arguments that all these solutions were easily scalable and all had a neutralization range of 1000m. These were the some of the most important aspects for this airport and all solutions with a matching higher than 60% also agreed to these specifications.<br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71308Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:47:47Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71307Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:47:34Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/ |drones.jortdebokx.nl] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71306Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:46:22Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
If any reader would be interested in reproducing these results, one could go to [https://drones.jortdebokx.nl/] (assuming it is still online) and fill in the propositions as we described in the section above. Alternatively, one could also try out some propositions, or could do a similiar approach as we did in this validation test.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71305Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:44:56Z<p>M.g.d.quincey@student.tue.nl: /* Conclusion of the validation for this airprot */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
We conclude that indeed the decision model passed this validation test. We saw that indeed all solutions that did not match the mandatory property of being easy to extend successully filtered out all solutions that did not match this criteria. Furthermore, we also saw that we indeed got solutions with a proper matching percentage, with some solutions being well over a 70% match. Furthermore, we also saw that it did not happen that a lot of solutions got the same score, which was something we were slightly worried about in the beginning. <br />
<br />
Furthermore, considering the solutions with a high match, we also concluded that these would fit the airport very well, as the specifications of those solutions seemed to allign very well with the beliefs and wants of our airport. With respect to the solutions that got a very low score, we saw that all these were indeed solutions that were either not a very good idea for any airport, or solutions that simply were not a good match with our airport in question. From this, we conclude that the decision model passed this validation test very well. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71303Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:40:24Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best seven solutions. We did this because the other solutions had a score lower than this, and thus we would argue that including and explaining all possible solutions, even those with a low score, would be superfluous for the validation. We can see that we have seven solutions, each with a proper matching percentage, ranging from 76.9% to 66.7%.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71302Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:38:25Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
As you can see, we have only listed the best five solutions.<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=File:Olaukodrones.jpg&diff=71301File:Olaukodrones.jpg2019-04-03T14:38:18Z<p>M.g.d.quincey@student.tue.nl: uploaded a new version of "File:Olaukodrones.jpg"</p>
<hr />
<div></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=File:Olaukodrones2.jpg&diff=71300File:Olaukodrones2.jpg2019-04-03T14:37:35Z<p>M.g.d.quincey@student.tue.nl: </p>
<hr />
<div></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71299Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:35:09Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: Picture of the best solutions for this airport from the decision model.]]<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=File:Olaukodrones.jpg&diff=71298File:Olaukodrones.jpg2019-04-03T14:33:44Z<p>M.g.d.quincey@student.tue.nl: </p>
<hr />
<div></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71297Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:33:19Z<p>M.g.d.quincey@student.tue.nl: /* Outcome of the decision model for this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
[[File:olaukodrones.jpg]]<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71296Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:30:54Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71295Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:30:43Z<p>M.g.d.quincey@student.tue.nl: /* Detection */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Agree.<br />
<br />
Motivation: With all airports being more environmentally friendly, we decide that we also want our airport to go in this direction. <br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71294Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:29:06Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
=== More important propositions ===<br />
There are some propositions which we find more important than others, so in the last window we checked the following propositions to be more important to us:<br />
<br />
Proposition 3: he detection system must detect illegal UAV presence within 1 second.<br />
<br />
Motivation: We find it very important for the anti-UAV system to be quickly deployable.<br />
<br />
<br />
Proposition 11: The detection system must be able to detect UAVs automatically. <br />
<br />
Motivation: We think that the automization is very important, because else we would constantly have to place precious personnel on the detection system.<br />
<br />
<br />
Proposition 13: The detection system must be able to operate under any weather condition.<br />
<br />
Motivation: We would have a huge weakness if we could only neutralize UAVs in good weather, then we could not use them for a significant amount of the time. <br />
<br />
<br />
Proposition 18: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
Motivation: We must be able to reach all parts of of our airport, because else we would again have a significant weakness. <br />
<br />
<br />
Proposition 20: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
Motivation: We want to ensure the safety of everybody at the airport, and when there can be UAVs falling down, we would consider this harmful for the passengers and our reputation.<br />
<br />
<br />
Proposition 25: The neutralisation system must be easy to extend.<br />
<br />
Motivation: Analogous to the scalability of the airport; since we want to extend the airport in the near future, the extendability of the neutralisation system is of great importance.<br />
<br />
<br />
Proposition 27: The neutralisation system must be able to <br />
neutralize drones under any weather circumstance.<br />
<br />
Motivation: See motivation of proposition 13.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71291Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:13:35Z<p>M.g.d.quincey@student.tue.nl: /* Detection */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box '''mandatory property'''.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agree.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agree.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71290Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T14:13:02Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71288Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:55:42Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
30. Q: The neutralisation system must be able to be moved around instead of a permanent installation.<br />
<br />
A: Agree.<br />
<br />
Motivation: With highly agile UAVs, we also want to be able to move the neutralization installation, since we think this is more cost-effective.<br />
<br />
<br />
31. Q: The neutralisation system must be able to be used without training of the employees.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although this is more desirable, we do not think it is possible to employ such a neutralization system without training some employees. <br />
<br />
<br />
32. Q: The neutralisation system must be able to operate in the event of a power outage.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same as why the detection system should not necessarily be able to operate in the event of a power outage.<br />
<br />
33. Q: The neutralisation system must be able to neutralize drones without human input.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unsure whether it would be ethical and 100% reliable to take out drones without human input; we should at least have somebody responsible over the neutralization.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71287Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:52:28Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to detect UAVs at night.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71286Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:52:06Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71285Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:51:20Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
17. Q: I want to be advised on a anti-UAV neutralisation solution.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to stand idly by when we have detected a drone. We also want to be able to neutralize it when we find it necessary.<br />
<br />
<br />
18. Q: The neutralisation system must be able to neutralise UAVs within a range of at least 1000m from the neutralisation system.<br />
<br />
A: Agree.<br />
<br />
Motivation: The range of this is about the range of our airport that we want to be able to actively defend, so this fits our airport well.<br />
<br />
<br />
19. Q: The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We consider a few minutes too long to wait; if there is something we must be able to intervene quicker than a few minutes.<br />
<br />
<br />
20. Q: The neutralisation system must not pose any threat to humans, for example when a drone falls from the sky after being neutralised.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want people to get hurt by our anti-UAV solution. <br />
<br />
<br />
21. Q: The detection system must not emit any CO2.<br />
<br />
A: Neutral.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
22. Q: The neutralisation system must be suitable to use in locations close to residential areas.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not located near residential areas, so we do not have to worry about that.<br />
<br />
<br />
23. Q: Almost all commercially available UAVs use predictable Radio Frequencies for communication, however, the neutralisation system must also be able to <br />
neutralise UAVs (possibly privately built) that use other communication methods.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as for the detection system.<br />
<br />
<br />
24. Q: The system must be able to neutralise a drone threat without destroying the drone in question.<br />
<br />
A: Agree.<br />
<br />
Motivation: In order to not create chaos, and to better find out who controlled the drone, we find it more convenient to be able to neutralize the drone without e.g. 'shooting it out of the sky'.<br />
<br />
<br />
25. Q: The neutralisation system must be easy to extend.<br />
<br />
A: Agree.<br />
<br />
Motivation: As described before, we want to scale up our airport, and hence the neutralization system must be easy to extend.<br />
<br />
<br />
26. Q: The neutralisation system must be able to neutralize swarms of drones simultaneously, rather than one at a time.<br />
<br />
A: Disagree.<br />
<br />
Motivation: The same reasoning as to why we do not deem it necessary to detect multiple drones concurrently.<br />
<br />
<br />
27. Q: The neutralisation system must be able to neutralize drones under any weather circumstance.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate under any weather circumstances.<br />
<br />
<br />
28. Q: The neutralisation system must be able to operate 24/7.<br />
<br />
A: Agree.<br />
<br />
Motivation: The same as why the detection system must be able to operate 24/7.<br />
<br />
<br />
29. Q: The neutralisation system must be able to neutralise UAVs at night.<br />
<br />
A: Agree.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71284Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:40:33Z<p>M.g.d.quincey@student.tue.nl: /* Neutralization = */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
=== Neutralization ===<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71283Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:40:02Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
=== Detection ===<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
== Neutralization ===<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71280Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:26:55Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable. We think this is so important that we will make this a critical proposition by checking the box.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71279Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:20:47Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: Agree.<br />
<br />
Motivation: We foresee that when the technology of radar techniques will change, then all solutions that rely on this radar technique will be useless. Hence, we consider it a '''critical aspect''' that the solution should not rely on radar techniques. <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71278Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:16:45Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Agree.<br />
<br />
Motivation: One second can be critical when it comes to drone encounters, and since we prioritize safety, we are willing to spend more to get a solution that operates quickly.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: <br />
<br />
Motivation: <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Agree.<br />
<br />
Motivation: We want and are able to expand more due to our location, and we have concrete plans to scale up our airport in the near future. Hence, we deem it necessary to make the solution scalable.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71277Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:13:24Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although we would like to have a quick detection system, we would argue that within one second is enormously fast. We are also okay with solutions that take a little longer, since we cannot afford an overly expensive solution.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: <br />
<br />
Motivation: <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unable to expand more due to our location, so we do not have any concrete plans to scale up our airport in the near future. Hence, we do not deem it necessary to make the solution scalable.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A: Neutral.<br />
<br />
Motivation: We have plenty of space, so we do not really care about the size of the solution. <br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Only the commercial drones can be properly identified. We foresee that the ones that seek to do harm will not be able to be identified, so this attribute is not important to us. <br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
<br />
A: Agree.<br />
<br />
Motivation: We do not want to have someone permanently occupied by detecting UAVs. The whole reasons of such a system was to have this automized.<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
<br />
A: Disagree.<br />
<br />
Motivation: In the event of a power outage, we will not fly our airplanes anyways and hence, we do not find it necessary to detect drones when there are no departures and arrivals. <br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Even in bad weather, we might fly our airplanes and then we must most certainly have a detection system, especially when they are not easily visible. <br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
<br />
A: Agreed.<br />
<br />
Motivation: We may have long days at the airport, and then we want the detection system to be working all the time. <br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
<br />
A: Agreed.<br />
<br />
Motivation: Although we do not have departing or arriving airplanes in the middle of the night, we certainly have airplanes departing or arriving very early in the morning, or very late in the evening. At those times, it is dark, and also then we want to be able to properly detect drones. <br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Since we are a small airport, we do not find it necessary to have the solution portable, as long as it has full range at its permanent installation spot.<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71276Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:07:24Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although we would like to have a quick detection system, we would argue that within one second is enormously fast. We are also okay with solutions that take a little longer, since we cannot afford an overly expensive solution.<br />
<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: <br />
<br />
Motivation: <br />
<br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unable to expand more due to our location, so we do not have any concrete plans to scale up our airport in the near future. Hence, we do not deem it necessary to make the solution scalable.<br />
<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
A:<br />
<br />
Motivation:<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71275Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T13:05:55Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
<br />
A: Agree<br />
<br />
Motivation: Because we as an airport are looking for a system that is able to detect UAVs.<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Our airport is not that big; we only need a range of about 1000m, which is much less than that.<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
<br />
A: Disagree.<br />
<br />
Motivation: Although we would like to have a quick detection system, we would argue that within one second is enormously fast. We are also okay with solutions that take a little longer, since we cannot afford an overly expensive solution.<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
<br />
A: Agree<br />
<br />
Motivation: We do not want our passengers or people living nearby to get annoyed by our anti-UAV solution; we are willing to invest a little in order to have a less loud solution. <br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
<br />
A: <br />
<br />
Motivation: <br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We are unable to expand more due to our location, so we do not have any concrete plans to scale up our airport in the near future. Hence, we do not deem it necessary to make the solution scalable.<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
<br />
A: Disagree<br />
<br />
Motivation: We are an airport that do not see a lot of drone encounters and hence, we estimated that the probability of having multiple drone encounters concurrently is extremely small. Hence, we decide not to invest into this solution. <br />
<br />
8. Q: The detection system must not emit any CO2.<br />
<br />
A: Disagree.<br />
<br />
Motivation: We as an airport decided that CO2 emission is of a lower priority than safety of the passengers and cost.<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
<br />
A:<br />
<br />
Motivation:<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
A:<br />
<br />
Motivation:<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
A:<br />
<br />
Motivation:<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
A:<br />
<br />
Motivation:<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
A:<br />
<br />
Motivation:<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
A:<br />
<br />
Motivation:<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
A:<br />
<br />
Motivation:<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
A:<br />
<br />
Motivation:<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71274Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T12:57:56Z<p>M.g.d.quincey@student.tue.nl: /* Filled in proposition on behalf of this airport */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. Q: I want to be advised on an anti-UAV detection solution. <br />
A: <br />
<br />
Motivation:<br />
<br />
2. Q: The detection system must be able to detect UAVs within a range of at least 4000m.<br />
A:<br />
<br />
Motivation:<br />
<br />
3. Q: The detection system must detect illegal UAV presence within 1 second.<br />
A:<br />
<br />
Motivation:<br />
<br />
4. Q: The detection system must not make any loud noises annoying people around the airport.<br />
A:<br />
<br />
Motivation:<br />
<br />
<br />
5. Q: Most detection systems make use of Radar techniques to detect unwanted UAVs, however, privately built drones can be made of materials such that they are not picked up by radar systems. Hence, the detection system should not only rely on Radar techniques for detection.<br />
A:<br />
<br />
Motivation: <br />
<br />
6. Q: The detection system must be able to scale with the growth of the airport in size.<br />
A:<br />
<br />
Motivation:<br />
<br />
7. Q: The detection system must be able to detect multiple UAVs concurrently.<br />
A:<br />
<br />
Motivation:<br />
<br />
8. Q: The detection system must not emit any CO2.<br />
A:<br />
<br />
Motivation:<br />
<br />
9. Q: The detection system must fit in an area of 0.5m^3.<br />
A:<br />
<br />
Motivation:<br />
<br />
10. Q: The detection system must be able to properly identify the UAV.<br />
A:<br />
<br />
Motivation:<br />
<br />
11. Q: The detection system must be able to detect UAVs automatically.<br />
A:<br />
<br />
Motivation:<br />
<br />
12: Q: The detection system must be able to operate in the event of a power outage<br />
A:<br />
<br />
Motivation:<br />
<br />
13: Q: The detection system must be able to operate under any weather condition.<br />
A:<br />
<br />
Motivation:<br />
<br />
14: Q: The detection system must be able to operate 24/7 (assuming no outages, et cetera take place).<br />
A:<br />
<br />
Motivation:<br />
<br />
15: Q: The detection system must be able to detect UAVs at night.<br />
A:<br />
<br />
Motivation:<br />
<br />
16: Q: The detection system must be able to be moved around instead of the solution being a `permanent’ installation.<br />
A:<br />
<br />
Motivation:<br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71273Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T12:50:53Z<p>M.g.d.quincey@student.tue.nl: /* Internal Validation */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
== Picking an airport to use for validation ==<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into account were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
== Filled in proposition on behalf of this airport ==<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. <br />
2.<br />
3.<br />
4.<br />
5.<br />
6.<br />
7.<br />
8.<br />
9.<br />
10. <br />
<br />
== Outcome of the decision model for this airport ==<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
== Discussion of the results for this airport ==<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
<br />
== Conclusion of the validation for this airprot ==<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71272Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-03T12:47:30Z<p>M.g.d.quincey@student.tue.nl: /* Internal Validation */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
== Validation by Domain Experts ==<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
== Internal Validation ==<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. Since we do not know all the needed information for a certain existing airport in the Netherlands, we decided to come up with our own mock-up airport. We know all the important attributes, beliefs and wants of this airport, which allows us to fill in the questionnaire on their behald. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to come up with this airport and the reasoning behind what to answer to which proposition. Examples of important attributes we took into accoutn were among other the size of the airport and the amount of daily departures and arrivals. <br />
<br />
So, when we created our mock-up airport, we came up with these answers to the propositions, with corresponding motivation: <br />
<br />
1. <br />
2.<br />
3.<br />
4.<br />
5.<br />
6.<br />
7.<br />
8.<br />
9.<br />
10. <br />
<br />
After filling the questionnaire, we filled the results of this questionnaire into our decision model. As expected, the decision model gave as output a list of anti-UAV solutions, together with a percentage score. These were the results:<br />
<br />
* insert picture of result *<br />
<br />
Now, we discussed with all the group members whether these outcomes did make sense. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Airports_under_a_microscope_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71172Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 32019-04-02T14:18:13Z<p>M.g.d.quincey@student.tue.nl: </p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Airports =<br />
<br />
== Commercial airports ==<br />
=== Introduction ===<br />
In this section, we will take a look at commercial airports and inspect the aspects of such airports that might affect our decision model for unwanted UAVs. Here, we define commercial airports as airports where the main purpose is to earn money by transporting passengers, transporting cargo loads or selling goods/services on the airport itself. We will first be looking at some general commercial airports in the Netherlands and compare these airports to find differences between them that might be critical for the decision model. After that, the USE stakeholders of commercial airports will be discussed. Furthermore, we will do a risk analysis of the types of drones that might be expected for this type of airport, which might be different for any type or airport. Lastly, we will look at the requirements that an anti-UAV system for commercial airports should adhere to. <br />
<br />
=== General ===<br />
In the Netherlands, there are quite a few commercial airports ranging from very large airports with almost 71 million passengers a year to smaller airports of 250 thousand passengers, or less, yearly <ref name="cbs data vliegvelden"></ref>. Larger airports might have a bigger budget to invest in an anti-UAV system. However, such airports might also be more prone to UAV hindrance or attacks. Next, to that, the requirements of an anti-UAV system also might be very dependent on the attributes of the airports, which could vary a lot between certain airports. Apart from moving around passengers, some commercial airports also move around huge amounts of cargo loads yearly <ref name="vliegveldinfo"></ref>, while others might not work at all. Amongst other things, the area of an airport, the number of runways at an airport or certain rules or regulations of an airport can all be a factor on deciding the right type of anti-UAV system for a certain airport. This is why it is so crucial to investigate different (commercial) airports that are located in the Netherlands.<br />
<br />
In the section, five of the commercial airports located in the Netherlands will be inspected briefly. We will not do a full analysis of these airports as we think this will not be of any help our decision model, but we will be looking at the attributes of the airports that might be interesting or be of any impact to our decision model. The commercial airports that will be looked into are:<br />
* Amsterdam Airport Schiphol<br />
* Eindhoven Airport<br />
* Rotterdam The Hague Airport<br />
* Rotterdam The Hague Airport<br />
* Maastricht Aachen Airport<br />
<br />
The reason for choosing these five particular airports is because they differ quite a lot from each other on different attributes, such as the size of the airport, the number of passengers transported yearly, the amount of cargo load being transported yearly, the layout of the airport et cetera. Furthermore, we feel like these five airports cover/generalise to most of the commercial airports located in the Netherlands and cover most of the different attributes that might impact the type of anti-UAV system being used.<br />
<br />
==== Amsterdam Airport Schiphol ==== <br />
Amsterdam Airport Schiphol is the largest airport of the Netherlands and was the third busiest airport of Europe in 2017. <ref name="vliegveldinfo">Vliegveldinfo[https://www.vliegveldinfo.nl/vlieginfo/grootste-vliegveld-europa/ "Grootste vliegveld Europa"] Retrieved on 2019-02-20.</ref> Schiphol is home to Dutch airlines KLM, Martinair, Corendon, TUI fly and Transavia. The airport has 6 runways with a total length of 19467 metres. <ref name="schiphol facts">Royal schiphol group[https://www.schiphol.nl/nl/route-development/pagina/amsterdam-airport-schiphol-airport-facts/ "Amsterdam airport schiphol airport facts"] Retrieved on 2019-02-20.</ref> In figure 1 an overview and the locations of the runways of Schiphol can be observed with the names and lengths of each runway included.<br />
[[File:Runways_schiphol.png| 650 px |thumb|upright=4|center|alt=Missing image|Figure 1: General overview of the layout of Schiphol and overview of the runways of Schiphol. <ref name="schiphol facts"></ref>]]<br />
<br />
<br />
Table 1 below contains general facts of the Amsterdam Schiphol Airport, mainly of 2018. <br />
<br />
{| class="wikitable" | style="vertical-align:middle;" | border="2" style="border-collapse:collapse" ;<br />
|+ '''Table 1: General facts of Amsterdam Airport Schiphol <ref name="schiphol facts"></ref><ref name="schiphol">Royal schiphol group [https://www.schiphol.nl/nl/schiphol-group/pagina/verkeer-en-vervoer-cijfers/ "Schiphol verkeer en vervoer cijfers"] Retrieved on 2019-02-20.</ref>'''<br />
|-<br />
| Passenger amount 2018 || 70.956.604 passengers<br />
|-<br />
| Amount of airplane movements (departure/arrival) in 2018 || 499.446 movements<br />
|-<br />
| Total cargo load 2018 || 1.716.982 tonne<br />
|-<br />
| Amount direct worldwide destinations || 322 destinations<br />
|-<br />
| Area of the airport || 2.787 hectares<br />
|-<br />
| Arrival peak hour capacity || 106 movements<br />
|-<br />
| Departure peak hour capacity || 110 movements<br />
|-<br />
| Amount runways || 6 runways<br />
|}<br />
<br />
<br />
In the table, we observe that Schiphol had a total of 499.446 movements in 2018, that would be an average of around 1.367 movements every day. Keep in mind that this is both arrivals from and departures on Schiphol and includes both passenger flights and cargo flights. Then with the six runways the airport has, this would mean that with an average of 1.367 movements every day, each runway would, on average, observe an aeroplane movement around every 6 minutes. Note that this is a calculated average, and this number of movements on each runway can be higher during peak hours. This, in turn, means that at any given point in time, Schiphol is extremely busy and loaded with at least thousands of passengers. Hence any form of interception with UAVs would already cause major issues. Let alone the damage that could be caused by any forms of terrorist attacks with the use of UAVs.<br />
<br />
==== Eindhoven Airport ====<br />
Eindhoven Airport is the second largest airport in the Netherlands in terms of passengers <ref name = "EindhovenWiki">[https://en.wikipedia.org/wiki/Eindhoven_Airport "Wikipedia: Eindhoven Airport"], Retrieved on 20-02-2019</ref>. Furthermore, Eindhoven Airport is also the home base of all military transport aeroplanes of the Royal Netherlands Air Force <ref name = "EindhovenLuchtmacht">[https://www.defensie.nl/organisatie/luchtmacht/vliegbases-en-luchtmachtonderdelen/eindhoven "Koninklijke Luchtmacht: Vliegbasis Eindhoven"], Retrieved on 20-02-2019</ref>. In total, there are nine large military aeroplanes, which are usually used after a natural disaster<ref name = "InfoGraph">[file:///C:/Users/s165048/Downloads/7183-klu-infographic-vlb-eindhoven-a3staand-v7.pdf "Infograph Vliegbasis Eindhoven 2017"], Retrieved on 20-02-2019.</ref>. The latest example being hurricane Irma, who caused huge amounts of damage in the Dutch colonies Sint Maarten and Curacao <ref name = "Irma">[https://en.wikipedia.org/wiki/Hurricane_Irma "Wikpedia: Hurricane Irma"], Retrieved on 20-02-2019. </ref>. <br />
<br />
In Figure 2 below, the general layout of the Eindhoven Airport is depicted, showing mainly the single runway the airport has and the parking spots of the aeroplanes currently not in use.<br />
<br />
<br />
[[File:eindhoven_airport.png| 650 px |thumb|upright=4|center|alt=Missing image|Figure 2: General overview of layout of Eindhoven Airport <ref name="eindhoven infrastructuur">NACO,EA Infra werkgroep [https://www.google.com/url?sa=i&source=images&cd=&ved=2ahUKEwig9MLQ2szgAhXFblAKHejfBZ0Q5TV6BAgBEAs&url=http%3A%2F%2Fsamenopdehoogte.nl%2Fover-ons%2Fnieuws%2Fairport-infrastructuur&psig=AOvVaw1LoO6la2qC5v9x5hVz6RDh&ust=1550834409875173 "Eindhoven Airport Master Plan<br />
Airport Infrastructuur"] May 2018, Retrieved on 21-02-2019.</ref>]]<br />
<br />
<br />
Table 2 below contains general facts of Eindhoven Airport, mainly of 2018. <br />
<br />
{| class="wikitable" | style="vertical-align:middle;" | border="2" style="border-collapse:collapse" ;<br />
|+ '''Table 2: General facts of Eindhoven Airport <ref name="cbs data vliegvelden">CBS opendata [https://opendata.cbs.nl/statline/#/CBS/nl/dataset/37478hvv/table?ts=1514376943049 "Luchtvaart; maandcijfers Nederlandse luchthavens van nationaal belang"] Retrieved on 2019-02-20.</ref>'''<br />
|-<br />
| Passenger amount 2018 || 6.237.755 passengers<br />
|-<br />
| Amount of airplane movements (departure/arrival) in 2018 || 38.642 movements<br />
|-<br />
| Total cargo load 2018 || 0 tonne<br />
|-<br />
| Amount direct worldwide destinations || 81 destinations<br />
|-<br />
| Area of the airport || 639 hectares<br />
|-<br />
| Amount runways || 1 runway<br />
|}<br />
<br />
<br />
As can be seen in the table, Eindhoven Airport has only one runway, which means that both the commercial aeroplanes and the military aeroplanes depart and arrive using the same runway <ref name = "Runway">[https://www.routesonline.com/airports/2415/eindhoven-airport/about/ ""Routes Online: Eindhoven Airport], Retrieved on 20-02-2019 </ref>. Eindhoven Airport is also experiencing an enormous growth at the moment, with an increase in served passengers of almost 10% <ref name = "Routes Online: Eindhoven Airport, Milestone Year 2018">[https://www.routesonline.com/airports/2415/eindhoven-airport/news/282530/milestone-year-2018/ "Routes Online: Eindhoven Airport, Milestone Year 2018"], Retrieved on 20-02-2019 </ref>. The military departures occur so infrequently compared to the commercial airlines that their contribution to the number of departures from Eindhoven Airport is negligible. The operating hours of Eindhoven Airport are between 07:00 in the morning and 24:00, so midnight, 365 days per year. This corresponds to an aeroplane either departing or arriving on the runway every 10 minutes on average<ref name = "Runway">[https://www.routesonline.com/airports/2415/eindhoven-airport/about/ ""Routes Online: Eindhoven Airport], Retrieved on 20-02-2019 </ref>.<br />
<br />
Some critical remarks regarding the safety hazards of Eindhoven Airport are that it is, in fact, the second largest airport in the Netherlands in terms of passengers. This means that a sudden shutdown due to a drone would have enormous consequences for many passengers. Furthermore, due to the fact that the airport only has one runway, it has a single point of failure to prevent all aeroplanes from both taking off and arriving. If one drone would be flying around this runway, then the whole airport has to be shut down. On top of that, the fact that Eindhoven Airport houses military transport vehicles increases the risk of drone attacks. Imagine that there would be a sudden natural disaster and these aeroplanes need to depart to aid the people in the disaster area, then it takes only one drone to delay this much-needed help by several hours. This could be the hours that would mean life or death for multiple people in the disaster area. Lastly, the giant and continuous growth of Eindhoven Airport could also pose problems in terms of drone safety. If the growth is not adequately regulated and safety measures are not adequately investigated, the growth could increase the possibility of a drone causing problems at Eindhoven Airport.<br />
<br />
==== Rotterdam The Hague Airport ====<br />
Rotterdam The Hague Airport is the third largest airport of the Netherlands with respect to served passengers served, with a total of 1.943.733 in 2018<ref name="rotterdam the hague airport">Rotterdam The Hague Airport [https://www.rotterdamthehagueairport.nl/content/uploads/2019/01/Totaal-per-jaar.pdf "Rotterdam The Hague airport, overzicht verkeer en vervoer per kalenderjaar"] Retrieved on 2019-02-20.</ref><ref name = "WikiHague">[https://en.wikipedia.org/wiki/Rotterdam_The_Hague_Airport "Wikipedia: Rotterdam The Hague Airport"], Retrieved on 20-02-2019.</ref>. As the name suggests, it is located between the Dutch cities Rotterdam and The Hague. Due to the airport being located close to the political heart of The Netherlands, Rotterdam The Hague Airport functions as the airport of the government <ref name = "Overheidsluchthaven">[https://www.rotterdamthehagueairport.nl/onderneming/overons/geschiedenis/ "Rotterdam The Hague Airport: Geschiedenis"], Retrieved on 20-02-2019.</ref>. Important international guests use this airport to go to the Netherlands, for example to the Nuclear Safety Summit in 2014 <ref name = "Overheidsluchthaven">[https://www.rotterdamthehagueairport.nl/onderneming/overons/geschiedenis/ "Rotterdam The Hague Airport: Geschiedenis"], Retrieved on 20-02-2019.</ref>.<br />
<br />
[[File:rotterdam_airport.png| 650 px |thumb|upright=4|center|alt=Missing image|Figure 3: General overview of layout of Rotterdam The Hague Airport. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 21-02-2019</ref>]]<br />
<br />
<br />
Table 3 below contains general facts of Rotterdam The Hague Airport, mainly of 2018. <br />
<br />
{| class="wikitable" | style="vertical-align:middle;" | border="2" style="border-collapse:collapse" ;<br />
|+ '''Table 3: General facts of Rotterdam The Hague Airport <ref name="cbs data vliegvelden">CBS opendata [https://opendata.cbs.nl/statline/#/CBS/nl/dataset/37478hvv/table?ts=1514376943049 "Luchtvaart; maandcijfers Nederlandse luchthavens van nationaal belang"] Retrieved on 2019-02-20.</ref><ref name = "Data">[https://www.rotterdamthehagueairport.nl/content/uploads/2018/07/feiten-en-cijfers-2017.pdf "Rotterdam The Hague Airport: Feiten en Cijfers 2017"], Retrieved on 20-02-2019.</ref>'''<br />
|-<br />
| Passenger amount 2018 || 1.943.733 passengers<br />
|-<br />
| Amount of airplane movements (departure/arrival) in 2018 || 53.322 movements<br />
|-<br />
| Total cargo load 2018 || 19 tonnes<br />
|-<br />
| Amount direct worldwide destinations || 50 destinations<br />
|-<br />
| Area of the airport || 222 hectares<br />
|-<br />
| Amount runways || 1 runway<br />
|}<br />
<br />
<br />
What is unique about this airport is that it has much more departing and arriving aeroplanes than Eindhoven Airport, whereas Eindhoven Airport serves three times as many passengers. The reason for this is the type of flights that occur at the airport. Rotterdam The Hague Airport also houses many flying schools. Here, people train their flying skills to acquire a flying permit as a hobby. These flying lessons are also the main contributor to the number of movements, with a total of 13.761 <ref name = "RotterdamTheHaguePassengers">[https://www.rotterdamthehagueairport.nl/content/uploads/2019/01/Totaal-per-maand-2018.pdf "Rotterdam The Hague Airport: Overzicht verkeer en vervoer 2018"], Retrieved on 20-02-2019. </ref>.<br />
<br />
From this information, we can deduct some safety hazards of Rotterdam The Hague Airport. It is quite a small airport with not a lot of passengers, which means that fewer passengers will be affected, should the airport be shut down due to a drone in the area. However, there is again only one runway. This results in a single point of failure to prevent all aeroplanes from both taking off and arriving. If one drone would be flying around this runway, then the whole airport has to be shut down. Furthermore, the airport also serves a lot of passengers who are internationally and politically important, which increases the risk of someone who wants to disrupt this politically important person by delaying his or her flight. What is also important to note is that there are a lot of flying lessons and recreational departures and arrivals. In general, these planes are a lot smaller and are flown by less talented pilots than commercial aeroplanes. This means that a drone would do more damage to the smaller aeroplane. Furthermore, the lesser talented pilot is more likely not to know the rules correctly, and would most likely handle more poorly in the event of a drone in the area. The consequences of a drone collision to a small aeroplane would, however, be smaller in terms of people affected by the incident.<br />
<br />
==== Maastricht Aachen Airport ====<br />
<br />
Maastricht Aachen Airport is one of the larger airports of the Netherlands with 274.986 passengers in 2018<ref name="cbs data vliegvelden"></ref>, however, this is really small compared to the number of passengers of Schiphol in 2018. The airport is located eight kilometres north of Maastricht. <br />
<br />
In Figure 3 below, the general layout of the Maastricht Aachen Airport is depicted, showing mainly the single runway the airport has.<br />
<br />
[[File:maastricht_airport.png| 650 px |thumb|upright=4|center|alt=Missing image|Figure 4: General overview of layout of Maastricht Aachen Airport. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 21-02-2019</ref>]]<br />
<br />
Table 4 below contains general facts of Maastricht Aachen Airport, mainly of 2018.<br />
<br />
{| class="wikitable" | style="vertical-align:middle;" | border="2" style="border-collapse:collapse" ;<br />
|+ '''Table 4: General facts of Maastricht Aachen Airport<ref name="cbs data vliegvelden"></ref><ref name="Maastricht">[https://www.maa.nl/bestemmingen/, "https://www.maa.nl/bestemmingen/"]</ref>'''<br />
|-<br />
| Passenger amount 2018 || 274.986 passengers<br />
|-<br />
| Amount of airplane movements (departure/arrival) in 2018 || 15.781 movements<br />
|-<br />
| Total cargo load 2018 || 124.676 tonnes<br />
|-<br />
| Amount direct worldwide destinations || 25 destinations<br />
|-<br />
| Area of the airport || 450 hectares*<br />
|-<br />
| Amount runways || 1 runway<br />
|}<br />
<br />
<nowiki>*</nowiki>Approximation of area calculated using a google maps area calculator <ref name="maps area calc">DaftLogic, [https://www.daftlogic.com/projects-google-maps-area-calculator-tool.htm#, "Google Maps Area Calculator Tool"], retrieved on 10-03-2019</ref><br />
<br />
==== Groningen Airport Eelde ====<br />
<br />
Groningen Airport Eelde is the fifth largest airport of the Netherlands in terms of passengers with 228.698 passengers in 2018<ref name="cbs data vliegvelden"></ref>, however, this is again really small compared to the number of passengers of Schiphol in 2018. The airport is located close to the village Eelde in Groningen. The airport has only one runway in use, the second (smaller) one depicted also in Figure 5 below, is not in use anymore.<br />
<br />
[[File:groningen_airport.png| 650 px |thumb|upright=4|center|alt=Missing image|Figure 5: General overview of layout of Groningen Airport Eelde. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 21-02-2019</ref>]]<br />
<br />
<br />
Table 5 below contains general facts of Groningen Airport Eelde, mainly of 2018.<br />
<br />
{| class="wikitable" | style="vertical-align:middle;" | border="2" style="border-collapse:collapse" ;<br />
|+ '''Table 5: General facts of Groningen Airport Eelde<ref name="cbs data vliegvelden"></ref><ref name="groningen website">[https://www.groningenairport.nl/bestemmingen-overzicht "https://www.groningenairport.nl/bestemmingen-overzicht"]</ref>'''<br />
|-<br />
| Passenger amount 2018 || 228.698 passengers<br />
|-<br />
| Amount of airplane movements (departure/arrival) in 2018 || 31.413 movements<br />
|-<br />
| Total cargo load 2018 || 0 tonnes<br />
|-<br />
| Amount direct worldwide destinations || 11 destinations<br />
|-<br />
| Area of the airport || 400 hectares*<br />
|-<br />
| Amount runways || 1 runway<br />
|}<br />
<br />
<nowiki>*</nowiki>Approximation of area calculated using a google maps area calculator <ref name="maps area calc">DaftLogic, [https://www.daftlogic.com/projects-google-maps-area-calculator-tool.htm#, "Google Maps Area Calculator Tool"], retrieved on 10-03-2019</ref><br />
<br />
==== Most important attributes ====<br />
The most important attributes of commercial airports. Where, with 'most important', we refer to the importance of the attributes of commercial airports concerning anti-UAV systems. If we look at the general facts of each airport, then the two main attributes that could influence the chosen type of anti-UAV system for a certain airport, are the size of the airport and the number of aeroplane movements yearly. The amount of runways an airport has is correlated to the size of an airport and the amount of passengers/cargo moved yearly is correlated to the amount of airplane movements yearly. As for size, an anti-UAV system should be able to cover the whole range of a massive airport (i.e. Schiphol airport), but for another relatively small airport (i.e. Maastricht Aachen airport) that same system might be disproportionate or too excessive and simply not affordable. On the other hand, the number of aeroplane movements, which is correlated to the number of passengers/cargo moved, might make an airport more/less susceptible to UAV attacks. Furthermore, the number of aeroplane movements of an airport might make the airport more/less susceptible for a UAV of a certain category to attack, which some anti-UAV systems could be specialised for. Since both attributes, size and amount of aeroplane movements, differ a lot between some of the commercial airports discussed above, these attributes are important to take into account for the decision model.<br />
<br />
=== USE stakeholders ===<br />
There are quite a number of stakeholders involved with commercial airports. Think about passengers, air carriers, shops at the airport, residents near the airport et cetera. All of these stakeholders have a goal for the airport and might have requirements for a potential anti-UAV system that such an airport might deploy. As a simple example, residents near the airport want to minimise any hinder or do not want to be affected in any way by the airport nearby, and therefore also want to minimise any hinder of an anti-UAV system deployed at the airport nearby. There are many more such stakeholders that want to minimise any hindrance an anti-UAV system could cause for their goals, concerning the airport. Therefore, an analysis of the stakeholders and their requirements for a potential anti-UAV system is important. In table 6, a list of stakeholders for commercial airports is given, including a definition or example of the stakeholder, and whether they belong to the user, society or enterprise group. <br />
<br />
<br />
{| class="wikitable" | style="vertical-align:middle;" | border="2" style="border-collapse:collapse" ;<br />
|+ '''Table 6: Stakeholders of commercial airports <ref name="airport stakeholders">David Schaar, [https://catsr.vse.gmu.edu/pubs/ICNS_Schaar_AirportStakeholders.pdf "Analysis of Airport Stakeholders"], The Volgenau School of Information Technology and<br />
Engineering, Retrieved on 05-03-2019</ref>'''<br />
! align="center"; style="width: 20%" | '''Stakeholder'''<br />
! align="center"; style="width: 5%" | '''Definition/Example(s)'''<br />
! align="center"; style="width: 5%" | '''USE'''<br />
|-<br />
| Passengers<br />
| Transferring passengers<br />
| User<br />
|-<br />
| Air carriers<br />
| Passenger and cargo carriers<br />
| Enterprise<br />
|-<br />
| General aviation users<br />
| Air taxi, corporate transportation, etc.<br />
| User<br />
|-<br />
| Airport organization<br />
| Management and staff with responsibility for operation of the airport<br />
| Enterprise<br />
|-<br />
| Investors and bond-holders<br />
| Individuals/companies investing in the airport or holding bonds with the airport<br />
| Enterprise<br />
|-<br />
| Concessionaires<br />
| Providers of services to passengers such as food or retail<br />
| Enterprise<br />
|-<br />
| Employees<br />
| Employees of the airport<br />
| Enterprise<br />
|-<br />
| Service providers<br />
| Providers of services to airport/air carriers such as fuel<br />
| Enterprise<br />
|-<br />
| Government<br />
| Responsible for infrastructure, security, etc.<br />
| Society<br />
|-<br />
| Communities affected by airport operations<br />
| Residents near the airport<br />
| Society<br />
|-<br />
| Ground transportation providers<br />
| Buses, shuttles, taxi's, rental cars, off airport parking services, etc.<br />
| Enterprise<br />
|-<br />
|}<br />
<br />
Each of these stakeholders has goals for the airport, and therefore a potential anti-UAV system must not hinder this goal. By listing the goals of the stakeholders, we can use these goals to set up requirements for anti-UAV systems for commercial airports later. Since most of the goals of the stakeholders with the same USE group overlap, we will summarise a list of the goals of each of the USE groups instead of every single stakeholder, to keep it comprehensible and easy to understand. This list is given below.<ref name="airport stakeholders"></ref><br />
<br />
==== Users goals for commercial airports ====<br />
* Moving passengers quickly and conveniently<br />
* Ensure all services are on-time<br />
* Keep fares as low as possible<br />
* Serve as an access point and ensure good availability and high equipment capability (for general aviation)<br />
<br />
==== Goals from enterprises for commercial airports ====<br />
* Ensure all services are on-time<br />
* Keep costs of services as low as possible<br />
* Maximise the number of passengers and cargo loads transported<br />
* Ensure safety of all services<br />
* Maximise the number of destinations served<br />
* Provide secure jobs and wages<br />
* Maximise users satisfaction<br />
* Minimise hindrance and noise for the area around the airport<br />
* Maximise environmental sustainability<br />
<br />
==== Goals from society for commercial airports ====<br />
* Minimise noise and hindrance for the area around the airport<br />
* Minimise emissions<br />
* Ensure airports can accommodate growth<br />
* Ensure safety and security of the airport<br />
<br />
This list shows a lot of the goals of each of the USE groups, which a potential anti-UAV system should not affect, or at least minimise any hindrance of those goals. These goals can translate to requirements for anti-UAV systems at commercial airports, which will be done later this section.<br />
<br />
=== Risk analysis ===<br />
Analysing the types of UAVs that are more likely to cause any hindrance/attacks for commercial airports can also form requirements for potential anti-UAV systems. As elaborated on in section [[drones - Group 4 - 2018/2019, Semester B, Quartile 3|drones]], there are different categories of UAVs, based on attributes of a UAV. Some anti-UAV solutions might not be able to accurately detect, identify and neutralise UAVs of a certain category. Therefore, might this category of UAVs be present at commercial airports, then it would not be well suited as an anti-UAV solution for this type of airport. On the other hand, some anti-UAV solutions might be specialised for a certain UAV category. If the majority of UAVs that are present at commercial airports are of this category, then this anti-UAV solution might be well suited for commercial airports. Therefore, an analysis of the types of UAVs that might form a threat to commercial airports is important as well, for forming the requirements for a good anti-UAV solution for commercial airports.<br />
<br />
Let us take the classes of UAVs C0 to C4 as discussed in section [[drones - Group 4 - 2018/2019, Semester B, Quartile 3|drones]] as the categories of UAVs we take into consideration. Starting with UAVs of class '''C0''', which is the only class of UAVs that do not require an electronic ID or geo-awareness. This means that the pilot of the UAV cannot easily be tracked, might the UAV fly over an area it is not supposed to. Furthermore, the UAVs in this class have not been programmed with geofencing to stay away from airports within a certain range. Hence, in theory, these UAVs are free to fly over any area the pilot wants. However, most of the (consumer) UAVs (at the moment of speaking, March 2019) in this class have a maximum controllable range of around 100-200 meters and a battery that lasts a maximum of about 15 minutes.<ref name="drones under 250 gram">Abbie, Outstandingdrone, [https://www.outstandingdrone.com/drones-under-250-grams/ "https://www.outstandingdrone.com/drones-under-250-grams/"], Retrieved on 05-03-2019</ref> So in practice, it is almost impossible for a UAV of this class to ever reach an airport and be of any hinder. Hence this class of UAV will probably not be much of a hindrance for commercial airports. Classes '''C1 to C4''' do require an electronic ID and geo-awareness. So, for any consumer bought UAV, they will be programmed with software that limits them such that these UAVs will not be able to enter a certain radius around the airport. So if consumers do not deactivate this software, then these classes of UAVs should also not be of any hindrance to commercial airports.<br />
<br />
Lastly, there is the class of privately build UAVs, for which the hobbyist builders themselves decide whether or not to implement any form of geo-fencing. This class of UAVs seems to be the most troublesome, as they are privately built, hence do not adhere to any rules necessarily. These drones also could be build of any weight and are not limited to reach a certain speed. Therefore it is not possible to assign them a drone class. Given that commercial airports, mostly, are hectic and crowded places, and might be a good target for anyone want to cause considerable damage to many people, these privately build UAVs could form a significant threat to commercial airports, especially if they are armed. Since a commercial airport would be such a good target for anyone that wants to cause a lot of harm/hindrance, it is also to be expected that privately build drones are very popular for these airports. So in short, the types of UAVs that can be expected at commercial airports are the ones of class C1 to C4, if their geo-fencing software were to be disabled, and privately build UAVs which could belong to any of the classes (except for C0). The drones of type C0 simply will not have enough range or enough battery life to cause hindrance to commercial airports.<br />
<br />
=== Requirements for solution ===<br />
After the investigation of multiple aspects on commercial, we have enough information to form requirements that an anti-UAV system should adhere to for commercial airports. We have discussed general facts of commercial airports, the stakeholders of these airports and what their goals are for the airports, and lastly a risk analysis of commercial airports. This should form a good base of were to be expected for an anti-UAV system at a commercial airport. The list of requirements below takes into account all found results from the investigation on commercial airports and translates them into the needed requirements for commercial airports.<br />
<br />
'''Requirements'''<br />
* Be able to take down drones in the entire area of the airport<br />
* Not interfere with an aeroplane at the airport<br />
* Take down a hostile drone within 3 minutes, to keep costs as low as possible and flights on time<br />
* Ensure safety of all services including passengers or any other humans at any time<br />
* Should not emit/minimise emission of CO2<br />
* Be able to be expanded to cover a bigger terrain in the future<br />
* Not be any louder than aeroplanes<br />
* Not hinder the user's satisfaction<br />
* Be able to take down any UAV from class C1 to C4<br />
<br />
== Military Airbases ==<br />
As of now, there are eight military bases for the Royal Dutch Air Force in the Netherlands \cite{royal}. However, they differ quite a lot in nature due to the housing of different types of aerial vehicles. There are main operation bases, which are the biggest bases. Furthermore, we have a tactical air operations base, which is the air traffic control centre that is used by all military air traffic, air battle management and air surveillance among others. Next, we also have defence helicopter command bases, where most of the attack helicopters are stationed. Then, we also have an air transport base, which is mostly used to transport either infantry or military equipment. Lastly, we also have a common support base, which is currently mainly used for military training purposes. Thus, the eight military bases can be structured as follows:<br />
<br />
* Main Operating Bases<br />
** Leeuwarden Air Base<br />
** Volkel Air Base<br />
* Tactical Air Operations Base<br />
** Air Operations Control Station Nieuw Milligen<br />
* Defense Helicopter Command Bases<br />
** Gilze-Rijen Airbase<br />
** De Kooy Airfield<br />
** Deelen Air Base<br />
* Air Transport Base<br />
** Eindhoven Air Base<br />
* Common Support Base<br />
** Woensdrecht Air Base<br />
<br />
=== Leeuwarden Air Base === <br />
Leeuwarden Air Base is together with the Volkel Air Base and the Gilze-Rijen Air Base on of the three biggest military air bases in the Netherlands \cite{leeuwarden_wikipedia_en}. It is one of the two bases that station the F-16s of the Royal Netherlands Air Force. It has two runways. The Leeuwarden Air Base is mostly used for homeland security when needed. Its main task is to actively monitor the airspace of the Benelux, deploying F-16s when necessary. At all times, Dutch F-16s are on stand-by. Apart from this monitoring, they are also deployed for missions all around the globe. Occasionally, they are also used for training sessions, which occur mostly above the North Sea \cite{leeuwarden_royal}.<br />
<br />
[[File:leeuwarden.JPG| 650 px |thumb|upright=4|center|alt=Missing image|Figure 6: General overview of layout of military air base Leeuwarden. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 08-03-2019</ref>]]<br />
<br />
=== Volkel Air Base ===<br />
Volkel Air Base is, as mentioned before, one of the biggest military air bases of the Rotal Netherlands Air Force \cite{volkel_wikipedia_en}. It is the other bases that stations the Dutch F-16s. With the housing of F-16s comes the task of providing air support and air defence. Just as done in Leeuwarden, the F-16s are most often deployed for monitoring the airspace of the Benelux \cite{volkel_royal}. It is located in the province of Noord-Brabant.<br />
<br />
[[File:volkel.JPG| 650 px |thumb|upright=4|center|alt=Missing image|Figure 5: General overview of layout of military air base Volkel. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 08-03-2019</ref>]]<br />
<br />
=== Air Operations Control Station Nieuw Milligen ===<br />
Air Operations Control Station Nieuw Milligen is one of the smaller military bases and is used mainly for air traffic control. At the moment, its main task is air traffic control \cite{milligen_wikipedia_en}. More concrete, this means for example that this air base checks (from their command-and-control centre) whether an aircraft adheres to its flight plan. If not, radio contact will be established from Nieuw Milligen. In the worst case, the command-and-control centre is capable of deploying tactical air control by scrambling a so-called Quick Reaction Alert. This means that two fully armed F-16 fighter aircraft will be deployed \cite{nieuwmilligen}. Note that this is only a control station and hence, no aerial vehicles are located at this air base. However, for the sake of completeness, we decided to include this air base in the report. <br />
<br />
=== Gilze-Rijen Air Base === <br />
Gilze-Rijen Air Base is a helicopter command base of the Royal Netherlands Air Force. These helicopters form a part of the Defence Helicopter Command \cite{gilze_wikipedia_en}. An example when helicopters from Gilze-Rijen are deployed as support for Navy ships. They can, if necessary, also be deployed as fighter helicopters \cite{gilze_royal}. In the case of a major fire, they can also be deployed. The air base is located in the south of the Netherlands. Like all the previously mentioned airports, Gilze-Rijen Air Base has two runways. Among the military aircraft, the Royal Air Force Historic Flight Foundation is located at the air base. Here, a collection of historic military aircraft is stored and occasionally operated. <br />
<br />
[[File:Gilze-Rijen.JPG| 650 px |thumb|upright=4|center|alt=Missing image|Figure 5: General overview of layout of Gilze-Rijen air base. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 08-03-2019</ref>]]<br />
<br />
=== De Kooy Airfield === <br />
De Kooy Airfield is a helicopter command base as well. It is located near Den Helder, a city in the Netherlands near the North Sea \cite{kooy_scramble}. Consequently, its main task is bringing workers from and to oil rigs located on the North Sea and also houses military helicopters \cite{kooy_wikipedia_en}. It only has one runway and is not used very often. <br />
<br />
[[File:De Kooy.png| 650 px |thumb|upright=4|center|alt=Missing image|Figure 5: General overview of layout of Airfield De Kooy. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 08-03-2019</ref>]]<br />
<br />
=== Deelen Air Base === <br />
Deelen Air Base is the last of the three helicopter command bases of the Royal Netherlands Air Force. It is located in Deelen, a small place in the middle of the Netherlands with about 50 inhabitants \cite{deelen_wikipedia_nl}. Its main task is for practices with helicopters. The main type of practices done in Deelen is communication and cooperation between on one side the helicopters and on one side the military on the ground. The military basis is also often used as a refuelling point for helicopters which are going on a mission abroad. Apart from fuel, ammunition is also loaded on the helicopters. The Air Base has only one runway.<br />
<br />
[[File:Deelen.png| 650 px |thumb|upright=4|center|alt=Missing image|Figure 5: General overview of layout of Deelen air base. <ref name="maps">Google [https://maps.google.com "Google Maps"] Retrieved on 08-03-2019</ref>]]<br />
<br />
=== Eindhoven Air Base ===<br />
Eindhoven Air Base is an air transport base, located in Eindhoven \cite{eindhoven_wiki}. As the type of air base suggests, all of the military transport vehicles are stationed here. In total, there are nine large military aerial vehicles, which are usually used after a natural disaster\cite{infograph}. The latest example being hurricane Irma, who caused huge amounts of damage in the Dutch colonies Sint Maarten and Curacao \cite{irma}. From the airport, goods and military were transported to help the people in need \cite{eindhoven_royal}. As described before, the military base uses the same single runway as the commercial aeroplanes. For an overview of the runway, see the section about Eindhoven Airport in the commercial aeroplanes section.<br />
<br />
=== Woensdrecht Air Base ===<br />
Woensdrecht Air Base is a common support base for the Royal Netherlands Air Force located in the south-west of the Netherlands. The airport has one runway. It is a base that is mainly used for training and logistics and does not house any combat military. As of now, the air base is not operational, meaning that no military aerial depart or arrive as of now from the air base, but for the sake of completeness, we want to mention the presence of this military base.<br />
<br />
== General description/facts of recreational Airfields in the Netherlands ==<br />
We distinguish a third main category of airport, which recreational airfields. These locations are often used by pilots-in-training, or for recreational flights. Traditionally meant for small, single prop planes like the well-known Cessna, these airfields often feature short runways. These runways are often simply flattened strips of grass or dirt since the types of aeroplanes that might take off, and land from these airfields do not have stringent requirements for takeoff. Often these airfields are home to one or more local sky-gliding organisations. It is common that also other organisations frequent these airfields, whether they are flight related organisations or not. It is noteworthy that these airfields are also frequently prime locations for drone flight. Being large, open spaces away from any residential areas, the aspects that make these places into great locations for recreational or educational flight, also make them suitable for drone flight.<br />
<br />
=== Airfield Malden ===<br />
<br />
Airfield Malden is an airfield mainly used for glide flights near Malden and Nijmegen in the Netherlands. It is situated on a large open field in the forests. In existence since 1954, Malden Airfield is mainly used by two glide flying organisations. These are the NijAC (Nijmeegse Aeroclub), the local glide aero sport club, and NSA Stabilo, which is catered towards students from local educational institutes. The Airfield is relatively small and features a reasonably popular restaurant. Given the location of Malden Airfield in the forest, there are many cycling routes and hiking trails that either start, end, or pass by Malden Airfield and thus by the restaurant. This, plus the entertainment provided by watching planes take off and land, makes Malden Airfield into a fairly popular location for not only flight related visitors but also cyclists, hikers and the general population.<br />
The Airfield in Malden has two runways. A fleet consisting of 9 glide planes including a motorised one is stationed at the Airfield. These planes belong to the NijAC as the main group behind the operations at this Airfield. As this Airfield supports mainly the flights of club members of the NijAC, no real full-time personnel is present. The NijAC offers many services to non-members, such as educational glide flying, flying as recreation with one of the club members, or offering their location as a service for non-members flying their plane. It should be mentioned that the services, popularity and reachability of Malden Airfield would make this into a perfect location to fly commercial drones. Larger drones or even RC (radio controlled) model planes could make use of the runways at this airfield, granted that safety is a priority.<br />
This information, together with the mentioned fact that this Airfield is frequently visited by many people with non-flight related goals, makes Malden Airfield a unique situation for an anti-drone system. Since Malden Airfield is a prime location for the practice of recreational flight, this might also include drone flight. Organised events like drone races might also attract more business for both the flying clubs and the restaurant. Drone flight is not necessarily something we wish to discourage here permanently, and might be lucrative for all parties involved if we can guarantee safety.<br />
<br />
=== Airfield Terlet ===<br />
<br />
Terlet Airfield is home to the most significant association of gliding flight clubs in the Netherlands, consisting of the following active clubs: Gelderse Zweefvlieg Club, Delftse Studenten Aeroclub, Gliding Adventures Europe, Kennemer Zweefvlieg Club, ZC Deeleen, the EZZC, ZHVC and Zweefvliegclub Ameland. The Airfield has six grass winch tracks, where winches might be used to tug glide planes up to take off speed. Small motorised planes might also be used to tug gliding planes, but only on 1 of the take-off/ landing tracks. Airfield Terlet is larger than the previously discussed Malden Airfield, averaging around 17.000 glide plane take-offs each year. Around 10% of which is done by using a motorised plane as a tug. This is important to note that since different types of planes might incur different kinds of risks for drone incidents. Unlike Malden Airfield, there are no popular cycling routes or hiking trails near the Airfield, and entering the grounds without the supervision of an experienced glide plane pilot or local instructor is strongly discouraged. Reasons stated for this are that the Airfield can be quite dangerous without proper supervision, as planes are taking off and landing and more importantly when a gliding plane is tugged using a ground-based winch, the cable used to attach the plane will fall to the ground after takeoff. This is very dangerous to the uninformed or inexperienced. On the website of the club association, it states that there is a camping ground nearby, meant for use by glide plane enthusiasts and club members. There is no mention of this Airfield being a popular location for drone enthusiasts or non-flight related activities. However, it should be noted that since motorised planes take off and land from Terlet Airfield, it is not a stretch to imagine larger UAVs using the runway as well. This could be a future expansion possibility, and then the question becomes how safety should be guaranteed.<br />
<br />
=== Possible risks ===<br />
<br />
We can conclude that non-flyers and non-flight related clubs or organisations are also often present at recreational airfields. The more people are present. The more people are at risk of a possible drone incident. To get a gliding aeroplane off the ground, often a motorised aeroplane is used to propel it forward by interlinking them. These motorised aeroplanes are often more susceptible to drone incidents. <br />
As mentioned before, the attributes of recreational airfields also make them excellent locations for civilian drone flight. Parents might take their children to test out a newly bought drone with a camera and take pictures. These airfields are often located in large, open fields and away from direct residential areas. As drone regulations include that direct line of sight must be maintained with the drone, these types of locations make it easier to follow these, and other, regulations. We conclude that recreational airfields are a right place for recreational drone flight, and thus are more prone for drone incidents including recreational drones.<br />
<br />
== Recreational Airfield ==<br />
<br />
=== Introduction ===<br />
This Section covers recreational airfields. These are defined as airfields, where the main purpose is not to earn money from airlines by enabling them to transport people as a service. Rather, the main purpose is to enable flight for people whose enjoyment is in the flight itself. Concrete examples are airfields where mainly sky-gliding planes are flown, or where pilots make private flights for enjoyment, usually in small aeroplanes. This type of airfield is so different from the commercial type airports discussed before, and they are open to different types of drone incidents involving different categories of drones.<br />
<br />
=== General ===<br />
In this Section, we look at the various recreational airfields throughout the Netherlands. We will research how these types of airfields influence the requirements for anti-drone solutions, by analysing airport-type specific risks and the perspectives of the stakeholders. Recreational airfields are a platform for recreational flight. Recreational flight is made up of various subcategories, such as flight in glide planes, pilots in training making their first flight hours in small aeroplanes, or simply hobbyist pilots flying their plane for fun. Often, the airfields that support this kind of flight are fairly small, and located in large open fields in forests, away from towns or cities. The characteristics of these locations that make them suitable for recreational flight, also make them good locations for drone flight. This will be elaborated on in the risk analysis at the end of this Section.<br />
<br />
====Malden airfield==== <br />
Malden airfield is an airfield mainly used for glide flights near Malden and Nijmegen in the Netherlands. It is situated on a large open field in the forests. In existence since 1954, Malden Airfield is mainly used by two glide flying organisations. These are the NijAC (Nijmeegse Aeroclub), the local glide aero sport club, and NSA Stabilo, which is catered towards students from local educational institutes. The Airfield is relatively small and features a fairly popular restaurant. Given the location of Malden Airfield in the forest, there are many cycling routes and hiking trails that either start, end, or pass by Malden Airfield and thus by the restaurant. This, plus the entertainment provided by watching planes take off and land, makes Malden Airfield into a fairly popular location for not only flight related visitors but also cyclists, hikers, and the general population.<br />
<br />
The general layout of the airfield is visible in the aerial photograph visible below.<br />
<br />
[[File:Malden_zweefvliegveld.png|400px|thumb|center|Malden airfield]]<br />
<br />
As visible in the figure above, the Airfield in Malden has two runways. A fleet consisting of 9 glide planes including a motorised one is stationed at the Airfield. These planes belong to the NijAC as the main group behind the operations at this Airfield. As this Airfield supports mainly the flights of club members of the NijAC, no real full-time personnel is present. The NijAC offers many services to non-members, such as educational glide flying, flying as recreation with one of the club members, or offering their location as a service for non-members flying their plane. <br />
It should be mentioned that the services, popularity, and reachability of Malden Airfield would make this into a perfect location to fly commercial drones. Larger drones or even RC (radio controlled) model planes could make use of the runways at this airfield, granted that safety is a priority.<br />
This information, together with the mentioned fact that this Airfield is frequently visited by many people with non-flight related goals, makes Malden Airfield a unique situation for an anti-drone system. Since Malden Airfield is a prime location for the practice of recreational flight, this might also include drone flight. Organised events like drone races might also attract more business for both the flying clubs and the restaurant. Drone flight is not necessarily something we wish to permanently discourage here and might be lucrative for all parties involved if we can guarantee safety.<br />
<br />
====Terlet Airfield==== <br />
Terlet Airfield is home to the largest association of gliding flight clubs in the Netherlands, consisting of the following active clubs: Gelderse Zweefvlieg Club, Delftse Studenten Aeroclub, Gliding Adventures Europe, Kennemer Zweefvlieg Club, ZC Deeleen, the EZZC, ZHVC, and Zweefvliegclub Ameland. The Airfield has six grass winch tracks, where winches might be used to tug glide planes up to take off speed. Small motorised planes might also be used to tug gliding planes, but only on 1 of the takeoff/ landing tracks. A map of the layout of Terlet Airfield is given below in Figure below.<br />
<br />
[[File:Terlet_map.png|400px|thumb|center|Terlet Airfield]]<br />
<br />
Airfield Terlet is larger than the previously discussed Malden Airfield, averaging around 17.000 glide plane take-offs each year. Around 10% of which is done by using a motorised plane as a tug. This is important to note since different types of planes might incur different kinds of risks for drone incidents.<br />
Unlike Malden Airfield, there are no popular cycling routes or hiking trails near the Airfield, and entering the grounds without the supervision of an experienced glide plane pilot or local instructor is strongly discouraged. Reasons stated for this are that the Airfield can be quite dangerous without proper supervision, as there are planes taking off and landing and more importantly when a gliding plane is tugged using a ground-based winch, the cable used to attach the plane will fall to the ground after takeoff. This is very dangerous to the uninformed or inexperienced.<br />
On the website of the club association, it states that there is a camping ground nearby, meant for use by glide plane enthusiasts and club members. There is no mention of this Airfield being a popular location for drone enthusiasts or non-flight related activities. However, it should be noted that since motorised planes take off and land from Terlet Airfield, it is not a stretch to imagine larger UAVs using the runway as well. This could be a future expansion possibility, and then the question becomes how safety should be guaranteed.<br />
<br />
=== USE Stakeholders ===<br />
As discussed previously, recreational airfields cater to a large variety of user groups, not limited to flight oriented users. As such, the variety of stakeholders in the safety of these airfields is equally large. This Section will identify such stakeholder groups and classify them according to the USE classification. <br />
<br />
{| class="wikitable" style="vertical-align:middle;" | border="2" style="border-collapse:collapse" ;<br />
! style="font-weight:bold; font-size:14px; font-family:Georgia, serif !important;;" | Stakeholder<br />
! style="font-weight:bold; font-size:14px; font-family:Georgia, serif !important;;" | Definition/Example(s)<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Recreational glide flying organisations<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Organisations using the airfield to offer glide flights to members<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Educational flight organisations<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Organisations offering lessons in flight of motorised airplanes<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Other sport organisations<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Cycling or hiking organisations that are also frequently present at airfields<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | People wanting to fly recreationally<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | The populace in general with the need for recreational flight<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | People needing transport<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | The need for people or goods being transported in small planes to land or<br />
take off at places other than the nations largest airports<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | People wanting to fly drones<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | People living close to the airfields wanting to fly their drone recreationally<br />
in suitable locations<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Employees<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Regular employees of the airfields<br />
|-<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Companies<br />
| style="font-size:14px; font-family:Georgia, serif !important;;" | Companies wanting to conduct business on the airfield by offering<br />
flights or other services<br />
|}<br />
<br />
In the table above, a list of stakeholders in drone flight safety at recreational airfields is introduced. The elements of this list all represent groups that have a stake in safety at recreational airfields, following from the analysis of several recreational airfields in the Netherlands. The interests of these stakeholders play a part in deciding which requirements might be more critical for an anti-drone system at a recreational airfield. Since recreational airfields are usually also prime locations for drone flight, the need to fly drones recreationally is counted as a society stake. The following list sums up the main goals of the various USE categories with regards to recreational airfields.<br />
<br />
We see that for the User category mainly safety is of importance. With regards to drone flight, it is essential for the User to be provided with levels of safety not inferior to that in a world or time where no drone flight was present at all. For society, it is of great interest to have these airfields available for flight as recreation, as well as transport to locations not close to major airports. However, it could also be described as a social goal to enable recreational flight with drones by maximising the usable airspace. Enterprise goals are mainly similar to the User goals in that safety is of great importance for companies with a stake in general operations at the airfield, however, maximising the area where drones can be flown safely is also a goal for drone manufacturing companies. In the following section, we further analyse the risk of drone incidents that are specific to this category of an airfield.<br />
<br />
=== Risk analysis ===<br />
Different types of airfields with different attributes might be at a higher risk of specific drone-related incidents than others. The most obvious elevated risk level exists because almost all recreational airfields are great places to fly drones. There are a number of non-enforced rules for commercial drone flight, one such being the ability to keep a line of sight to the flying drone at all times during operation. When commercial drone flight gets more popular and more affordable, we expect to see the number of such flights rise sharply in large open spaces away from residential areas especially because these rules are easier to adhere to. This is an important aspect for most commercial drone pilots. Other aspects of recreational airfields that might appeal to pilots of 'over-the-counter' drones might be the ease of access to most recreational airfields and the fact that there are often restaurants or cafes on site, altogether making recreational airfields attractive for a commercial visit for other purposes than to fly planes.<br />
<br />
Given this expected increase in the presence of commercial drone flight, it is pivotal that an anti-drone system at recreational airfields is able to maintain safety mainly when faced with smaller drones that are commercially available. One might use the argument that these commercially available drones might soon be equipped with a geofencing system, therefore solving a large part of the problem. However, as drone flight is also technically a form of recreational flight, a recreational airfield might not want to permanently prevent drones from flying through its airspace. As described in the USE stakeholder analysis, society, in general, might be a stakeholder in keeping many suitable areas, including these airfields, open for drone flight. Furthermore, according to the drone classification into C0 up to C4 categories, the smaller drones in the C0 category are not required to be equipped with geo-awareness. As described in the Section on commercial airports, many commercially available drones from this category have an operating range not exceeding 200 meters and a battery-time of about 15 minutes. While this might not be enough range nor usage time to cause serious trouble at a commercial airport, it might certainly be enough to cause an incident on a recreational airfield.<br />
<br />
It is also important to note that since recreational airfields are less busy, and the drone category we expect to see most is also that which has the least potential to cause damage due to being relatively small, general risks of drone incidents are reduced compared to major airports. Also, the subjective cost of such an incident might be less than that of an incident at a major airport, since a crash of a smaller plane endangers less human lives, and they are in less danger compared to an incident involving a large jet. Combined with the relatively low number of flights leaving from and arriving at these airfields, this all might suggest a lower risk of drone incidents at recreational airfields and might be seen as a reason to not completely inhibit drone flight here.<br />
<br />
=== Requirement for the solution ===<br />
This Section describes a list of requirements for an anti-drone system to serve as a recreational airfield. These were generated after considering airfield specific drone incident risks and the USE stakeholder analysis for these types of airfields. Note that some of these focus on minimising damage to the drone, as drone flight is also regarded as a type of recreational flight. Also, general environmental concerns are taken into account.<br />
<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
<br />
==References==<br />
<references /></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71171Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-02T14:09:03Z<p>M.g.d.quincey@student.tue.nl: /* Internal Validation */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
=== Role-playing === <br />
Testing the credibility of the model through role-playing will be done by proposing an example scenario where the individual is a higher-up at an airport company. Here, this individual will decide on what mechanisms to consider when it comes to illegal drone activity. This will be done based on certain information given to the individual. The individual will be asked for what they initially think the best solution is for the airport against unwanted UAVs. Then, the individual will be asked to fill in the questionnaire, report on their result, and describe what they think of the proposed solution. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
= Validation by Domain Experts =<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
= Internal Validation =<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
So, as described, our approach was as follows. We picked a specific airport and filled in the questionnaire on their behalf. We used the information from our [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3| airport analysis]] to fill in certain propositions which required this knowledge. Examples are the size of the airport and the amount of daily departures and arrivals. After filling the questionnaire, we filled the results of this questionnaire into our decision model, and looked at the outcomes. Together, we discussed whether these results do indeed make sense. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71170Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-02T13:12:58Z<p>M.g.d.quincey@student.tue.nl: /* Internal Validation */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
=== Role-playing === <br />
Testing the credibility of the model through role-playing will be done by proposing an example scenario where the individual is a higher-up at an airport company. Here, this individual will decide on what mechanisms to consider when it comes to illegal drone activity. This will be done based on certain information given to the individual. The individual will be asked for what they initially think the best solution is for the airport against unwanted UAVs. Then, the individual will be asked to fill in the questionnaire, report on their result, and describe what they think of the proposed solution. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
= Validation by Domain Experts =<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
= Internal Validation =<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
<br />
hier nog iets meer van hoe we t concreet gedaan hebben<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71169Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-02T13:12:41Z<p>M.g.d.quincey@student.tue.nl: /* Internal Validation */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
=== Role-playing === <br />
Testing the credibility of the model through role-playing will be done by proposing an example scenario where the individual is a higher-up at an airport company. Here, this individual will decide on what mechanisms to consider when it comes to illegal drone activity. This will be done based on certain information given to the individual. The individual will be asked for what they initially think the best solution is for the airport against unwanted UAVs. Then, the individual will be asked to fill in the questionnaire, report on their result, and describe what they think of the proposed solution. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
= Validation by Domain Experts =<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
= Internal Validation =<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. Over the last eight weeks, we have done an extensive literature research on the matter, and thus we also consider ourselves as people who can validate the model. As described before, we would have also let domain experts at Eindhoven Airport help us with this, but unfortunately their promise was not met. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=71168Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-04-02T13:08:47Z<p>M.g.d.quincey@student.tue.nl: </p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
=== Role-playing === <br />
Testing the credibility of the model through role-playing will be done by proposing an example scenario where the individual is a higher-up at an airport company. Here, this individual will decide on what mechanisms to consider when it comes to illegal drone activity. This will be done based on certain information given to the individual. The individual will be asked for what they initially think the best solution is for the airport against unwanted UAVs. Then, the individual will be asked to fill in the questionnaire, report on their result, and describe what they think of the proposed solution. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
= Validation by Domain Experts =<br />
As described before, we have sent the questionnaire above to the higher-ups at Eindhoven Airport that have the responsibility of the anti-drone systems. In our correspondence, we were assured that if we sent the questionnaire, we would get the feedback only a couple of work days later. Thus, we did send the questionnaire to this group of domain experts. Unfortunately, we did not receive the feedback during the duration of the course, and thus we were not able to analyze the feedback of the domain experts. Although it is unfortunate, we did learn an important lesson that relying on external sources can be unpredictable at times. <br />
<br />
= Internal Validation =<br />
So, in order to still provide a proper validation, we also did the validation internally, by all of the group members. <br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=70985Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-03-30T19:21:10Z<p>M.g.d.quincey@student.tue.nl: /* Methods */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.pdf | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
=== Role-playing === <br />
Testing the credibility of the model through role-playing will be done by proposing an example scenario where the individual is a higher-up at an airport company. Here, this individual will decide on what mechanisms to consider when it comes to illegal drone activity. This will be done based on certain information given to the individual. The individual will be asked for what they initially think the best solution is for the airport against unwanted UAVs. Then, the individual will be asked to fill in the questionnaire, report on their result, and describe what they think of the proposed solution. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=File:Questionnaire.pdf&diff=70984File:Questionnaire.pdf2019-03-30T19:20:50Z<p>M.g.d.quincey@student.tue.nl: Questionnaire of PRE 2018 3 Group 4.</p>
<hr />
<div>Questionnaire of PRE 2018 3 Group 4.</div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Decision_Model_validation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=70983Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 32019-03-30T19:19:32Z<p>M.g.d.quincey@student.tue.nl: /* Questionnaire */</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Decision Model Validation = <br />
== Introduction ==<br />
When introducing a decision model, it is important to both validate and verify that decision model. This is especially important when it comes to computational models. When it comes to model verification, we ask ourselves the following question: `Does the model perform as intended?'. This question is asked in order to verify that, for example, the model has been programmed correctly. Furthermore, it verifies if the algorithm has been implemented properly and if the model does not contain errors, oversights, or bugs. We also have model validation. Here, we ask ourselves the following question: `Does the model represent and correctly reproduce the behaviors of the real world system?'. Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained. The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate information about the system being modeled, and to makes the model actually used<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>.<br />
<br />
== What now? ==<br />
Unlike physical systems, for which there are well-established procedures for model validation, no such guidelines exist for social modeling. Unfortunately for the implemented decision model, there is no easy or clear way to validate and verify the model. This is mainly due to the model containing much subjectivity through human decision making. When users of the decision model use it, they have to provide input themselves. These inputs are not just numbers, but they are about whether or not the user agrees or disagrees with a proposition. This makes it hard to both validate and verify the model in a traditional way. In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model. Verification and validation work together by removing barriers and objections to model use. The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that `stand-in' for comparing model results to data from the real system<ref name="v&v">Model Verification and Validation, Charles M. Macal http://jtac.uchicago.edu/conferences/05/resources/V&V_macal_pres.pdf</ref>. This process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use. In order to still do some verification, we use subject matter experts in order to gain a grasp of the credibility of the model. We implement ways to measure this credibility through evaluation and role-playing.<br />
<br />
== Credibility ==<br />
As coined earlier, we want to somehow make the credibility of the model tangible. We do this through evaluation and role-playing. A group of domain experts will do the evaluation. These domain experts consist of both the group working on this project and higher-ups that go over anti-drone mechanisms at Eindhoven Airport. We asked higher-ups at Eindhoven Airport that go over anti-drone mechanisms to spread the decision model questionnaire and have it be filled in by numerous individuals that all agree on the interests, needs, and characteristics of Eindhoven Airport. Furthermore, we ask for an initial solution that they think is the best from the list of all the solutions we forged. It is then interesting to see if these individuals get the same results for the decision model and if they agree with the decision model. Additionally, it is interesting to compare the initial solution they thought would be best for the recommended solution they got and what they think of the recommended solution. Are they surprised? Are they not surprised at all? Does the recommended solution provide new insights?<br />
<br />
As we do not want to depend on a select few individuals from Eindhoven Airport alone, we also propose an example scenario where the user taking the questionnaire becomes a higher-up of a clearly defined airport that has to design a mechanism against unwanted UAVs. This is the role-playing method to establish credibility. This includes the needs, wants, and beliefs of this airport. We, internally, take this questionnaire as well. Afterward, we compare the initial thought of solutions, the recommended solutions, and the opinion of the recommended solution for the proposed airport. <br />
<br />
== Methods == <br />
Let us consider the two methods coined earlier for testing the credibility of the decision model to a certain degree.<br />
<br />
=== Evaluation ===<br />
Testing the credibility of the model through evaluation will be done, as briefly introduced earlier, by domain experts filling in a questionnaire that incorporates the decision model. We have sent a questionnaire to higher-ups at Eindhoven Airport that go over mechanisms to counter illegal drone activity around their airport. Additionally, we fill in this questionnaire ourselves from the perspective of Eindhoven Airport. This questionnaire first asks for the initial thought of the best solution from the list of solutions proposed. Then, the individual uses the decision model to obtain a recommended solution. Afterward, the opinion of the individual will be asked. Does the individual think this solution was to be expected? Does the solution make sense when holding it against the values and beliefs involved? What we are particularly interested in with this way of verification is seeing how much credibility we can give the recommended solutions based on the values and beliefs used for the input. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
The questionnaire we propose can be observed below. <br />
<br />
==== Questionnaire ====<br />
This file presents a questionnaire that takes into consideration questions that are used in the decision model. The goal of this decision model is to propose a solution for unwanted UAV presence around any type of airport. The primary goal of this questionnaire, that considers propositions, is to get feedback on the questions and the result of the model. This questionnaire is the basis of the decision model that we have implemented in order to recommend solutions against unwanted UAVs for stakeholders such as commercial airports and recreational airfields. Note that throughout this questionnaire, we use the point of view of Eindhoven Airport. That is, all propositions should be answered with the needs, wants, and ideals of Eindhoven Airport in mind. We address a multitude of propositions in the questionnaire, as well as provide context and motivation for these propositions. The motivation and context provided with each proposition are mainly for support and explanation of the proposition.<br />
<br />
We have decided to split the questionnaire into propositions that consider the two main types of anti-UAV solutions, namely detection, and neutralisation. On the one hand, the propositions that consider a solution for detection only provides a means to alert the airport of the presence of a UAV. On the other hand, the propositions that consider a solution for neutralisation only provides a means to take down the UAV once detected. Note that this questionnaire only considers the first draft of propositions and that this might change later on.<br />
<br />
For each proposition, the individual taking the questionnaire has to indicate to what extent they agree with the proposition. The options presented are `disagree’, `neutral’, and `agree’. The individual can indicate which option they choose by putting an `X’ in the respective cell. This system is used rather than a 5-point scale system as only an indication of what the solution has to offer is needed. Furthermore, it is incredibly complicated to divide solutions into various scales when compared to when considering two main groups.<br />
<br />
This questionnaire also has a PDF-format, which can be found [[Media:Questionnaire.PDF | here]]. <br />
<br />
'''General questions'''<br />
<br />
We first consider some general questions in order to process this feedback to improve the current decision model and the questions involved. <br />
<br />
* What do you personally think are the best solutions and why when it comes to detecting unwanted UAVs in the airspace?<br />
* What do you personally think are the best solutions and why when it comes to neutralising unwanted UAVs in the airspace?<br />
* How useful do you think a framework is that can give an indication on what kind of solution fits the needs, wants, and ideals of an airport. Note that this is not only meant for commercial airports, but also for recreational, and military ones.<br />
<br />
'''Detection'''<br />
<br />
'''1. I want to be advised on an anti-UAV detection solution'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Need for a solution<br />
<br />
Explanation: Because of the two different types of anti-UAV solutions, we decided to give the user the possibility only to pick one of either two types. Of course, it is still possible to be recommended for both types of solutions. This is done by agreeing to this proposition and the same proposition in the neutralisation questionnaire.<br />
<br />
Motivation: Certain small airports may decide due to budget constraints only to invest in detecting solutions, and merely to wait for the unwanted UAV to go away. Furthermore, certain airports which already have a decent neutralisation solution and do not want to invest in that again may only opt for a detection system.<br />
<br />
'''2. The detection system must be able to detect UAVs within a range of 4000 meters'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Range<br />
<br />
Explanation: The solution must work as described in the area inscribed by a circle with a radius of 4000m, centered at the detecting part of the solution.<br />
<br />
Motivation: The range has an enormous influence on the cost of the solution, which the user most likely wants to minimize, while also having a proper solution. For small airports, there is no immediate need to have a solution that covers three times the area of the airport. For larger airports, a solution that only covers half of the area is also not a favourable option.<br />
<br />
'''3. The detection system must detect illegal UAV presence within less than 1 second'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Speed of Operation<br />
<br />
Explanation: The time between the unwanted UAV entering the range of the anti-UAV solution, and the actual detection, must be less than one second.<br />
<br />
Motivation: The timing of detecting unwanted UAVs can be crucial at certain airports where security is a top priority, such as military airports. However, for some airports, the timing must be done quickly, but not close to instant.<br />
<br />
'''4. The detection system must not make any loud noises annoying people around the airport'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance of the environment<br />
<br />
Explanation: Certain solutions can emit a constant sound during operation, which could be an annoyance to people at or around the airport. Furthermore, some neutralisation solutions can also cause quite a loud noise when they are being operated.<br />
<br />
Motivation: The annoyance of people can be a less crucial factor in very remote airports with few passengers, such as military bases. However, at large airports with lots of (easily frightened) passengers, one might refrain from solutions which make loud noises.<br />
<br />
'''5. The detections system must be able to detect UAVs from all the categories(C1-C4)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Effect on Different Types of UAVs<br />
<br />
Explanation: There are different types of commercial UAVs, ranging from C1 being very small UAVs, to C4 being large and heavy UAVs. Some solutions can be very effective on smaller UAVs, but the larger UAVs may require more costly solutions.<br />
<br />
Motivation: Smaller recreational airports may decide only to be able to detect or neutralise smaller UAVs, since neutralising larger UAVs can result in more expensive solutions. If an airport concludes from investigations that they will most likely never encounter the larger C4 UAVs, then they can opt for a solution that only takes down the smaller UAVs.<br />
<br />
'''6. The detection system must be able to scale with the growth of the airport in size'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Scalability<br />
<br />
Explanation: When an airport grows in terms of size due to economic prosperity, the solutions must be able to easily expand with the growing airport. Some detection solutions, for example, can be more easily scaled by adding another small subpart, whereas other solutions may require adding a whole new unit as if you have two systems.<br />
<br />
Motivation: Some airports have already planned to grow and extend over the coming ten years. However, some airports have already reached their cap, meaning that they know that they will not scale up in the coming decade. For these airports, it is not wise to spend extra on solutions that have invested research into making their solutions more scalable.<br />
<br />
'''7. The detection system must be able to detect multiple UAVs concurrently'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Number of Drones it Can Handle<br />
<br />
Explanation: Some solutions can handle multiple drones concurrently. On the other hand, some solutions (such as an aimed jammer), can only be aimed at one UAV. Then, only one UAV can be detected or neutralised at the same time.<br />
<br />
Motivation: There are smaller airports that argue that the probability of two drones causing a disturbance at the same time is highly unlikely. Especially when saving costs, it might be wise to not spend extra money on more expensive solutions that can handle multiples UAVs concurrently.<br />
<br />
'''8. The detection system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Emission<br />
<br />
Explanation: Some solutions can be powered by fossil fuel, meaning that they emit CO2.<br />
<br />
Motivation: The transition to green energy can be the main priority for airports, whereas the emission of CO2 can be of much less importance for other airports who care less about these regulations.<br />
<br />
'''9. The detection system must not be larger than 1 m3'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Size<br />
<br />
Explanation: A solution is a physical object, which takes up a particular space. Some solutions are much more compact than other solutions.<br />
<br />
Motivation: Some airports may be small and not have enough space to have specific solutions that take up too much space.<br />
<br />
'''10. The detection system must be able to identify the UAV properly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Identification<br />
<br />
Explanation: Regulated drones also emit an identification signal, from which for example the product code and links to the owner can be enclosed. This proposition states that the solution is able to not only detect but also identify drones that emit these identification signals.<br />
<br />
Motivation: Although not all drones emit these signals, some airports may find it worth the cost to be able to identify these drones.<br />
<br />
'''11. The detection system must be able to detect UAVs automatically without needing any human interaction'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Autonomy<br />
<br />
Explanation: For specific solutions, a certain extent of human interaction is needed in order for the detection system to operate. This proposition puts a constraint of the detection system not requiring any form of human interaction. <br />
<br />
Motivation: In some instances where 24/7 protection is needed, it might be useful not to need any human interaction when it comes to the services provided by the detection system. This is especially useful since human interaction only requires more effort that could potentially result in errors being introduced.<br />
<br />
'''12. The detection system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Power Outage<br />
<br />
Explanation: This proposition states that the detection system must be able to operate after there has been a power outage. This can be through various ways, such as the detection system making use of a battery.<br />
<br />
Motivation: For some airports, it is vital that even after a power outage, the detection system still functions. It is, however, also possible that this is not a significant issue.<br />
<br />
'''13. The detection system must be able to operate under any weather condition'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Weather<br />
<br />
Explanation: This proposition states that the detection system must be able to detect UAVs under any weather condition. This means that UAVs should be detected even when there are hazardous conditions.<br />
<br />
Motivation: Some individuals might not want to put this constraint upon the solution as UAVs might not be able to fly under certain hazardous conditions.<br />
<br />
'''14. The detection system must be able to operate 24/7 (assuming no outages, et cetera take place)'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the solution providing 24/7 coverage when it comes to the detection of the UAVs in the airspace around the airport within a certain distance.<br />
<br />
Motivation: For some airports, it might be essential that there is 24/7 coverage because there are flights 24/7. For other airports, this might not be as important as they do not consider flights 24/7.<br />
<br />
'''15. The detection system must be able to detect UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Time<br />
<br />
Explanation: This proposition focuses on the constraint that UAVs should not merely be detected at daytime, but also at nighttime.<br />
<br />
Motivation: Certain airfields (recreational) where only flights are active at certain times during a week with set hours are not as interested in solutions that provide their services 24/7. Then, for these instances, it is attractive to consider solutions that contain fewer constraints due to this relieving the costs of the solution.<br />
<br />
'''16. The detection system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Portability<br />
<br />
Explanation: An airport can have the preference of a solution being portable. With this, we mean that it is possible for this solution to be `picked up’ and deployed elsewhere. This results in the airport being able to deploy the solution almost anywhere in their area while not having to invest in a solution that covers the whole area by itself.<br />
<br />
Motivation: Certain airports might not require a fully automated system that is active 24/7 due to financial constraints. Then, it is possible that they are interested in a less expensive solution that does not need to be active 24/7. Considering a portable solution is then an option. This solution can then be deployed when needed.<br />
<br />
'''Neutralisation'''<br />
<br />
'''1. The neutralisation system must be able to neutralize UAVs within a range of 1000m from the neutralisation system'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''2. The neutralisation system may neutralise unwanted UAVs within a few minutes rather than instantly'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''3. The neutralisation system must not pose any threat to humans, for example when a UAV falls from the sky after being neutralised'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Danger to Humans<br />
<br />
Explanation: Some solutions, such as lasers, damage a UAV mid-air, meaning that it will most<br />
likely fall to the ground. Other solutions, however, do not have this issue.<br />
<br />
Motivation: Crowded airports may want to invest money in order to minimize the danger to humans. However, other airports where there are much less passengers, the risk is also lower and hence, airports may decide not to spend too much money on this.<br />
<br />
'''4. The neutralisation system must not emit any CO2'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''5. The neutralisation system must be suitable to use in locations close to residential areas'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Disturbance to the Environment<br />
<br />
Explanation: Some solutions are less conservative than other solutions. For example, some solutions can cause great harm to others when misused, which is especially harmful when the airport is close to any residential areas.<br />
<br />
Motivation: Some airports that are located in a crowded area might be looking for solutions that cause less danger to the immediate environment, whereas airports that are located in practically the middle of nowhere do not have to worry about this.<br />
<br />
'''6. The neutralisation system must be able to neutralise non-commercial UAVs, those that might not be regulation conforming'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''7. The neutralisation system must be able to neutralise commercial UAVs'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''8. The neutralisation system must be easy to extend'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''9. The neutralisation system must be able to neutralise swarms of UAVs simultaneously, rather than only being able to deal with a single UAV at a time'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''10. The neutralisation system must be able to neutralise UAVs under any weather circumstance'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''11. The neutralisation system must be able to operate 24/7'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''12. The neutralisation system must be able to neutralise UAVs at night'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''13. The neutralisation system must be able to be moved around instead of the solution being a `permanent’ installation'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''14. The neutralisation system must be able to be used without training of the employees''' <br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
Category: Level of Training<br />
<br />
Explanation: Some solutions are much more complex than others, and require a significant extra training course for the employees that operate these solutions. On the other hand, some <br />
other solutions are much easier to use.<br />
<br />
Motivation: Smaller airports who do not want to invest in the extra training hours may want a solution that does not take a lot of training, especially when it is only one employee who needs to be trained. Furthermore, airports where there are a lot of part-time employees might suffer more from having to train all these people.<br />
<br />
'''15. The neutralisation system must be able to operate in the event of a power outage'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''16. The neutralisation system must be able to neutralise UAVs without human input'''<br />
<br />
* Agree<br />
* Neutral<br />
* Disagree<br />
<br />
This proposition has been explained and motivated in the section for detection.<br />
<br />
'''Closing questions'''<br />
<br />
It is important to obtain feedback and to use this appropriately in order to improve the current decision model and its questions.<br />
<br />
* What is your opinion on the different categories used for the propositions? Were they diverse enough or not at all? Is a certain category that you expected missing?<br />
* What is your opinion on the propositions proposed? Were they diverse enough or not at all? Is a certain proposition that you expect missing?<br />
* Other remarks<br />
<br />
Thank you for filling in this questionnaire.<br />
<br />
=== Role-playing === <br />
Testing the credibility of the model through role-playing will be done by proposing an example scenario where the individual is a higher-up at an airport company. Here, this individual will decide on what mechanisms to consider when it comes to illegal drone activity. This will be done based on certain information given to the individual. The individual will be asked for what they initially think the best solution is for the airport against unwanted UAVs. Then, the individual will be asked to fill in the questionnaire, report on their result, and describe what they think of the proposed solution. We then collect all the information and analyse it by comparing the results provided to one another. This will then be used for assessing the credibility of the model.<br />
<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references/></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Specific_problem_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=70979Specific problem - Group 4 - 2018/2019, Semester B, Quartile 32019-03-30T19:16:34Z<p>M.g.d.quincey@student.tue.nl: Undo revision 70939 by M.g.d.quincey@student.tue.nl (Talk)</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Specific problem =<br />
In this section, we elaborate on the specific problem we want to consider.<br />
Comparing the specific problem to the general problem, the specific problem considers a particular context and environment.<br />
This was done in order to limit the scope of the problem. <br />
<br />
== Specific problem description ==<br />
As described in the `Approach' Section, we expected the societal issue of unwanted UAV presence to be divisible into multiple subcategories. Following our initial literature study, we indeed found this to be the case. There are many axes along which the problem space can be divided. For example, we might consider a division based on the nature of the cause of a drone incident, and as such whether it was caused by human failure or technical failure. Another possible distinction can be given based on the specific part of society that is impacted, whether it be the privacy of individuals when a camera-equipped UAV flies over their backyard, or the safety of a group of users when there are UAVs present around an airfield. When we consider the existing legal regulations, another commonly occurring division is that between human-controlled and autonomous UAVs. <br />
<br />
This realisation leads us to formulate a more specific problem definition with a smaller scope. In our study, we consider possible deterrents against unwanted UAV presence around airports. This includes studying the current legal regulations considering UAVs, both in general and more explicitly considering airports. The term UAV is also divisible into multiple subcategories; for this study, we take all sub-types of UAV into account. These specific sub-types will be further discussed in the following Section. <br />
<br />
As can be observed in the image below, the number of drone-related incidents has risen dramatically over the last few years. The reason for this is that technology and its evolution moves faster than regulators, whose job is to maintain safety standard when confronted with ever-evolving aspects of technology. Regulations require extensive research into the technology it should apply to, and these regulations also take time to roll out. In the meantime, the technology in question does not stop evolving and by the time regulations take effect, the technology in question has often already evolved beyond the scope of the regulations. Different categories of drones, which will be further discussed later, pose different threats to aeroplanes. One might hit the windscreen of an aeroplane, posing a direct danger to the pilots and therefore the plane's passengers, or a drone might get sucked into the air stream entering the plane's engines and ultimately destroy a turbojet motor propelling the plane.<br />
<br />
[[File:Airprox Drones.JPG|thumb|upright=4|400px|right|alt=Missing image|Figure 1. Airprox reports involving drones and other objects.]]<br />
<br />
Because different types of drones and different types of airports exist, there are multiple types of incidents to consider. All of these are factors that should be taken into consideration when deciding which drone countermeasure should be applied in a particular situation. This means that anyone drone countermeasure is doubtful to work in a majority of situations, and care should be taken when choosing which countermeasure to invest in. A 'geo-fencing' system, which prevents commercial drones from entering certain no-fly zones, might be bypassed or disabled, not correctly implemented by the third party drone manufacturer, or include a multitude of other problems. A drone that launches a net to disable other drones might be difficult to operate and has many downtimes after it fails to take out another drone. When the safety standard is high, the ones at airports are exceptionally so; this becomes a difficult problem. It is not a simple decision to choose which drone countermeasure should be applied.<br />
<br />
== Examples of financial consequences ==<br />
<br />
It is essential to consider the financial consequences of the problems proposed in the specific problem description.<br />
Let us review a recent example that sparked the controversy regarding drone interception even more.<br />
The drone activity that obstructed flights in and out of London’s Gatwick airport for 33 hours cost airlines an estimated £50 million ($64.5 million)<ref name="gatwickairportje">Hallie Detrick, Gatwick's December Drone Closure Cost Airlines $64.5 million http://fortune.com/2019/01/22/gatwick-drone-closure-cost/</ref>.<br />
This estimate of £50 million is based on EasyJet’s disclosure that it lost £15 million ($19.3 million) in revenue and customer welfare combined during the 33 hours long illegal drone activity.<br />
Easyjet further stated that the drone incident was a wake-up call for airports. <br />
Not only Gatwick but also other airports are now plotting to try to enhance its response to any similar threats that may occur in the future.<br />
Gatwick’s flight interruptions affected about 140,000 people, where 82,000 of them were EasyJet customers.<br />
<br />
In July 2017, Dubai International Airport was shut down temporarily due to illegal drone activity. <br />
The costs of the shutdown were roughly $100,000 a minute according to Emirates Authority for Standardisation and Metrology (ESMA) estimates<ref name="dubaiairpotje">Shutting down Dubai International Airport due to a drone costs $100,000 a minute https://www.arabianbusiness.com/content/375851-drone-costs-100000-minute-loss-to-uae-airports</ref>.<br />
ESMA has introduced new regulatory standards for commercial and recreational use of drones, which includes a monitoring system for detecting UAVs in the country. <br />
In 2017, the Dubai Civil Aviation Authority (DCAA) head of airspace safety, Michael Rudolph, told Arabian Business that they were planning on testing their indigenously developed spectrum analysis technology to track threats of rogue drones and pinpoint their locations. With this, it would be possible to attack the rogue drones. <br />
In Dubai, it is now obligatory for all drone operators to apply for a license and undergo a training program.<br />
Since the illegal drone incidents, several new no-fly zones have been introduced by the General Civil Aviation Authority. <br />
<br />
In January 2019, Heathrow Airport had closed its runway after a possible drone sighting. <br />
This happened three weeks after the Gatwick fiasco, in which the airport was closed for 36 hours after multiple drone sightings had occurred. <br />
Around 140,000 travellers were impacted after 1,000 flights were cancelled or diverted.<br />
No numbers regarding the costs have been given, but we can only assume that these costs were relatively high but not as high as the estimated cost of Gatwick's incident. All in all, these are just a few examples of how tremendous the financial consequences of illegal drone activity around airports can be.<br />
<br />
== State of the Art ==<br />
Again, in order to gain more insights regarding the specific problem, we add additional information to the [[State_of_the_Art_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | State of the Art]] that was started for the general problem description. We further build upon it in this section. <br />
<br />
== Specific USE aspects == <br />
In this section, we consider the users, society, and enterprise when considering the specific problem description.<br />
<br />
=== Users ===<br />
When we take a look at the users in the more specific context of considering solutions against unwanted UAVs at and around airports, we see a shift in the types of users compared to the previously defined users in the more general setting. We see that mainly non-governmental organisations such as airports and airlines are the users of this more specific setting instead of the government, which was one of the users in our general setting. These types of users are also part of the use aspect `enterprise' and will be elaborated on in more detail in the corresponding section below. Indirectly, also passengers of flights are users of solutions against unwanted UAVs at and around airports, as passengers benefit from such solutions since they just want to be able to travel without any hindrance. Again passengers of flights is a type of users that overlaps with the use aspect 'society' and will be elaborated on in more detail in the section below. Lastly, companies with the intent/goal to intercept or detect flying objects such as UAVs are, to some extent, users of solutions against unwanted UAVs.<br />
<br />
=== Society ===<br />
Let us take a closer look at how this specific problem description is relevant when we consider a society concerning airports.<br />
If a UAVs enter the air space of an airport, aeroplanes are not allowed to land at the airport nor are they allowed to leave the airport.<br />
Therefore, airports have placed a ban on the usage of UAVs around the airport in order to make sure that aeroplanes can still land and leave the airport.<br />
As one might already know, a tremendous number of people visit airports every day. <br />
In 2017, Schiphol airport, located in Amsterdam, already counted 68 515 425 passengers<ref name="Schiphol">Traffic and transport figures https://www.schiphol.nl/en/schiphol-group/page/transport-and-traffic-statistics/</ref>.<br />
One can already imagine how enormous the consequences can be if this airport cannot be used for a few days. <br />
This means that if a UAVs flies by for whatever reason, a large number of people will not be able to travel.<br />
This results not only in many angry travellers but also in airport companies that have to compensate these travellers for the delays introduced by these UAVs. <br />
<br />
From a societal perspective, this would mean that all travellers have a risk of their flight being delayed, which is undesirable due to many reasons as elaborated on previously. <br />
As of now, we have only considered the situation where a UAV simply flies by but what if this UAV has malicious intentions. <br />
For example, what if this UAV has been weaponised and is used by terrorists or a specific individual with malicious intentions and is used to wreak havoc at the airport. <br />
Then, these weaponised UAVs could be extremely dangerous as they could result in mass-killings.<br />
This would be a colossal disaster, and this should be avoided at all cost. <br />
A disaster is not only bound to happen when we consider weaponised UAVs. <br />
A disaster could also occur when the systems of the airport do not detect one of these UAVs. <br />
This UAV could then end up damaging the aeroplane, which could result in perilous situations.<br />
For example, we can expect disastrous situations when a UAV gets stuck in the motor of an aeroplane. <br />
All things considered, UAVs cannot only be hazardous to society when operated by malicious attackers, but they can also introduce many annoyances.<br />
<br />
=== Enterprise ===<br />
When we restrict the USE case analysis to only deterrents against unwanted UAVs or drones around airports, the enterprise aspect of the use analysis becomes more concrete. The main type of enterprise that is under risk is the airport branch. The total revenue of the aviation industry in 2018 alone is a staggering 821 billion USD<ref name="Aviation revenue">[https://www.iata.org/pressroom/facts_figures/fact_sheets/Documents/fact-sheet-industry-facts.pdf "International Aviation Transport Industry: Fact Sheet"], December 2018. Retrieved on 11-02-2019.</ref>, so there are huge amounts of money at risk here. The current protocol is to suspend all flights of the airport by 30 minutes, the average lifespan of a drone. This means that, should drones occur often enough, all flights will be suspended for an indefinite amount of time<ref name="CNN drone protocol">[https://edition.cnn.com/2018/12/20/uk/drone-gatwick-airport-explainer-gbr-intl/index.html "CNN: How can a drone bring an airport to a standstill?"], December 2018. Retrieved on 11-02-2019.</ref>. This will, of course, have huge costs for multiple branches of the aviation industry. The three most notable branches in our opinion are the airports, the airlines and the companies who use aviation to transport goods. For these three enterprise branches, we will analyse the consequences of such an unwanted drone near an airport.<br />
<br />
==== Airports ====<br />
Airports suffer the largest loss in the case of such a drone in the airspace, which makes sense since it is where the problem is located. The airports suffer huge financial losses mainly through three different ways. First and foremost, no profits can be made when no planes fly. For example, the drone incident at Gatwick Airport caused a loss of over 50 million English Pounds<ref name="Independent Gatwick Cost">[https://www.independent.co.uk/travel/news-and-advice/gatwick-drone-airport-cost-easyjet-runway-security-passenger-cancellation-a8739841.html "The Independent: Gatwick drone disruption cost over £50 million"], January 2019. Retrieved on 11-02-2019.</ref>. This was caused by suspending all flights, which were just over 400, over the total duration of 33 hours. The airport suffers a huge blow to their reputation. So the hourly financial losses are huge for airports when flights must be suspended due to drones in the area. Furthermore, should one airport consistently suffer from the presence of unwanted drones, both airlines and travellers might opt to choose a different airport. The other airport might be farther away, but it will be a more reliable airport. This would result in a drop in total passengers at the airport.<br />
<br />
==== Airlines ====<br />
Airlines such as RyanAir, KLM or EasyJet also suffer huge losses during the event of a drone suspending or cancelling flights. The airlines are the companies actually offering flights to travellers. What also causes more losses for airlines is that they have to compensate the travellers for the delay or cancellation of their flights. By European Law, airlines are required to provide travellers with enough food, drinks, and nightly accommodations for as long as necessary<ref name = "European Aviation Laws">[https://europa.eu/youreurope/citizens/travel/passenger-rights/air/faq/index_nl.htm "European Aviation Laws"], November 2018, Retrieved on 11-02-2019</ref>. For a large sudden suspension of flights at one place, which is the case in our problem, this also becomes an enormously difficult task for an airline. If the airline would not act accordingly, the airline could also suffer from a huge reputation loss, resulting in travellers not flying with that airline anymore. Furthermore, the travellers are also eligible for financial compensation by European Law<ref name = "European Aviation Laws">[https://europa.eu/youreurope/citizens/travel/passenger-rights/air/faq/index_nl.htm "European Aviation Laws"], November 2018, Retrieved on 11-02-2019</ref>. <br />
Airports.<br />
<br />
Another aspect of this branch are the employees of the airlines. Apart from the company as a whole, the employees, such as flight attendants and pilots, suffer significantly from such an airline 'shutdown'. This is because of the way that the employees get paid. They do get a baseline salary, but the most salary they receive come from the hours that they are actually in the plane either flying or aiding passengers<ref name = "Aviation Employers">[https://www.huffpost.com/entry/why-flight-attendants-hate-delays-more-than-you-from_b_577d3f1de4b0746f5648b920 "Huffpost: Why flight attendants hate delays more than you"], July 2016, Retrieved on 11-02-2019</ref>. If the planes do not fly, they suffer from a huge salary cut, which means that the whole branch of airline employees have financial losses.<br />
<br />
==== Companies who transport goods via aviation ====<br />
Another enterprise that suffers from delaying and cancelling of flights at an airport are transport companies. In general, these goods are transported with different aeroplanes than passenger aeroplanes, but cargo is usually transported with passengers in the same aeroplane <ref name = "Cargo Transport">[https://www.cruisinaltitude.com/15-reasons-next-flight-may-delayed/ "Loyalty Travels: Why Do Flights Get Delayed – 15 Reasons Why Your Next Flight May Be Delayed"], 2018, Retrieved on 12-02-2019</ref>. Besides, cargo aeroplanes also frequently fly to airports just for packages, since restaurants, shops, e.g. are not necessary there, decreasing airport costs. However, these airports can also be subjected to an unwanted drone in the airspace. Again, the protocol is that the aeroplane will not land and thus will be either delayed or cancelled. This can have dire consequences for such companies. The delay of their goods usually set off a chain reaction of consequent delays, which can be devastating if the timing is crucial. In conclusion, the consequence of these delays for these companies is huge financial losses and tremendous logistic issues to fix the delays of their goods.<br />
<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references /></div>M.g.d.quincey@student.tue.nlhttps://cstwiki.wtb.tue.nl/index.php?title=Present_situation_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3&diff=70978Present situation - Group 4 - 2018/2019, Semester B, Quartile 32019-03-30T19:16:25Z<p>M.g.d.quincey@student.tue.nl: Undo revision 70940 by M.g.d.quincey@student.tue.nl (Talk)</p>
<hr />
<div><div style="font-family: 'Georgia'; font-size: 15px; line-height: 1.5; max-width: 800px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 30px;"><br />
<br />
<div style="display: block; position: absolute; right: 6%;"><br />
; Page navigation<br />
# [[PRE2018_3_Group4 | Root]]<br />
# [[Notes - Group 4 - 2018/2019, Semester B, Quartile 3|Notes from meeting]]<br />
# [[Initial ideas - Group 4 - 2018/2019, Semester B, Quartile 3|Initial ideas]]<br />
# [[Project setup - Group 4 - 2018/2019, Semester B, Quartile 3|Project setup]]<br />
# [[General problem - Group 4 - 2018/2019, Semester B, Quartile 3|General problem description]] <br />
# [[State of the Art - Group 4 - 2018/2019, Semester B, Quartile 3|State of the Art]]<br />
# [[Specific problem - Group 4 - 2018/2019, Semester B, Quartile 3|Specific problem description]]<br />
# [[Present situation - Group 4 - 2018/2019, Semester B, Quartile 3|Present situation]]<br />
# [[Drones - Group 4 - 2018/2019, Semester B, Quartile 3|Drone analysis]]<br />
# [[Solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Solution analysis]]<br />
# [[Airports under a microscope - Group 4 - 2018/2019, Semester B, Quartile 3|Airport analysis]]<br />
# [[Types of Decision Models - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model investigation]]<br />
# [[Decision Model - Group 4 - 2018/2019, Semester B, Quartile 3 | Decision Model implementation]]<br />
# [[Decision Model validation - Group 4 - 2018/2019, Semester B, Quartile 3|Decision Model validation]]<br />
# [[Categorizing solutions - Group 4 - 2018/2019, Semester B, Quartile 3|Categorising solutions]]<br />
# [[Web_Application_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | Web Application]]<br />
# [[Future - Group 4 - 2018/2019, Semester B, Quartile 3|Future]]<br />
# [[Conclusion - Group 4 - 2018/2019, Semester B, Quartile 3|Conclusion]]<br />
# [[Discussion - Group 4 - 2018/2019, Semester B, Quartile 3|Discussion]]<br />
</div><br />
<br />
= Present situation =<br />
In this section, we consider the present situation regarding the specific problem description. <br />
We interview an airport and look at current solutions.<br />
<br />
== Airport Interview ==<br />
In order to get a more unobstructed view of the issues our users (airports) face today we decided to ask them a couple of questions. We want to obtain a clear picture of their current approach to airport security regarding drones, what the consequences would be if a drone were to fly in their airspace right now, and what the consequences were of the 19th of December Gatwick incident. We will then ask them what their requirements would be for a drone defence mechanism.<br />
<br />
We asked the following questions:<br />
<br />
* What is the airport's current mechanism for detecting drones?<br />
* How will the airport respond when the drone is sighted in restricted aerospace?<br />
* Roughly how much damage will the airport take if a drone were to restrict air traffic for 1 hour?<br />
* The 19th of December and 21st of December drone attack at Gatwick airport caused over 1000 flights to be affected, did your airport get affected by the knock-on effects?<br />
* What would be the maximum budget for an automated anti-drone mechanism?<br />
* What kind of system would you imagine when thinking of anti-drone mechanisms? <br />
<br />
We contacted most major Dutch airfields; Eindhoven, Schiphol, Maastricht Aaken, Groningen, Twente, Den Helder, Rotterdam the Hague and Bergen op Zoom.<br />
<br />
Eindhoven airport responded to the questions, firstly stating that Eindhoven airport uses the runway and infrastructure provided by the Military airbase Eindhoven. This means that the Dutch Royal Airforce is responsible for air traffic control and hence the safety in the airport's airspace. We had the following answers to the aforementioned questions:<br />
<br />
<br />
'''What is the airport's current mechanism for detecting drones?''' <br />
<br />
At the moment the airport has no automated system to detect drones. At the moment this done by sight from the air traffic control tower.<br />
<br />
'''How will the airport respond when the drone is sighted in restricted aerospace?'''<br />
<br />
This depends on the location of the drone. At the moment an incident affecting air traffic has not yet occurred. When a drone is spotted we will suspend all traffic<br />
<br />
'''Roughly how much damage will the airport take if a drone were to restrict air traffic for 1 hour?''' <br />
<br />
I cannot answer this question [in detail], for the military activities, the impact will be limited. However, the impact on Eindhoven Airport will be much larger.<br />
<br />
'''The 19th of December and 21st of December drone attack at Gatwick airport caused over 1000 flights to be affected, did your airport get affected by the knock-on effects?''' <br />
<br />
We were not affected as there are no flights to Gatwick from Eindhoven<br />
<br />
'''What would be the maximum budget for an automated anti-drone mechanism?''' <br />
<br />
None, for safety there will always be a budget available.<br />
<br />
'''What kind of system would you imagine when thinking of anti-drone mechanisms?''' <br />
<br />
The location, altitude and flight-profile are crucial. The weight of a drone is also very important.<br />
<br />
<br />
The correspondent also told us he was very interested in our research, offering the opportunity for further collaboration.<br />
<br />
== Solutions ==<br />
<br />
In this section, we will take a look at solutions against unwanted UAVs at and around airports that are currently/in the near future being used by airports/authorities. These solutions might exclude many solutions that might be useful but are simply not in use due to for example the jurisdiction not being up to date with the current technology. However, a list of all possible solutions including solutions that might not even be feasible right now, but maybe within the next few years will be discussed in the section [[Solutions_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | solutions]]. It is important to note that there are different rules for different types of [[Drones_-_Group_4_-_2018/2019,_Semester_B,_Quartile_3 | drones]].<br />
<br />
* There will be European rules and regulations in the near future, expected around June 2019, obligating operators wanting to fly with a drone that is heavier than 250 gram to be registered. Drones will be obligated to send out identification signals such that authorities, for example, the police, are able to trace and identify the operator of the drone<ref name="remco dijkstra drones"> drs. C. van Nieuwenhuizen Wijbenga. [https://www.rijksoverheid.nl/onderwerpen/drone/documenten/kamerstukken/2019/01/15/beantwoording-vragen-van-het-lid-remco-dijkstra-vvd-over-drones-bij-londen-gatwick "Beantwoording vragen van het lid Remco Dijkstra (VVD) over drones bij Londen-Gatwick"], Ministerie van Infrastructuur en Waterstaat, 15 January 2019, Retrieved on 14-02-2019 </ref>.<br />
* With these same rules and regulations drones will be obliged to be equipped with geofencing software. This will restrict the operator to be able to fly close to an airport<ref name="remco dijkstra drones"></ref>.<br />
* Anti-drone systems deployed at two London airports are capable of tracking the devices from as far as six miles away. As well as being able to sever communications with the operator, some models can also destroy the drones using a laser beam. However, it is not exactly been released to the public as to what equipment is used and how it works<ref name = "invest"></ref>.<br />
* The police trains eagles to make them consider unwanted UAVs as preys, such that they would catch the UAVs and place them in a safe area. However, the Dutch police have already stopped using this solution because training the eagles is more expensive and complicated than they anticipated<ref name="eagles drones"> Thuy Ong. [https://www.theverge.com/2017/12/12/16767000/police-netherlands-eagles-rogue-drones "Dutch police will stop using drone-hunting eagles since they weren't doing what they're told"], 12 December 2017, Retrieved on 14-02-2019 </ref>.<br />
* In May of 2018, London Southend Airport successfully tested an anti-drone system that combines optical sensor and radio frequency to detect drones<ref name="gatwick unprepared"> Adam Bannister. [https://www.ifsecglobal.com/drones/anti-drone-tech-exists-gatwick-airport-utterly-unprepared/ "With anti-drone tech on the market, why was Gatwick Airport so unprepared?"], December 21 2018, Retrieved on 14-02-2019 </ref>.<br />
* The US Federal Aviation Authority trialled the Anti-UAV Defense System (Auds) system in 2016. It uses high powered radio waves to disable drones, it blocks their communication with the controller and switches them off mid-air<ref name="gatwick unprepared"></ref>.<br />
<br />
== Limitations ==<br />
<br />
=== The jurisdiction regarding drones is not up to date with current technology ===<br />
As is often the case, the laws we have are not able to keep up with the tremendous advancements of technology <ref name = "A">[https://www.technologyreview.com/s/526401/laws-and-ethics-cant-keep-pace-with-technology/ "MIT Technology Review: Laws and Ethics Can’t Keep Pace with Technology"], Written by V. Wadhwa, April 2014, Retrieved on 12-02-2019</ref>. This has happened many times already in history, for example with the rise of copyright laws at the end of the 19th century. Due to the huge advancements in copying and spreading literature, originals authors lost lots of money to people selling the author's work without proper permission. This was facilitated due to the rise in printing technologies. Under the pressure of this growing technology, the copyright laws had been created, albeit years and years later after the problem had occurred <ref name = "B">[https://digitalcommons.law.scu.edu/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=1022&context=facpubs "Does Technology Require New Law?"], Written by D. Friedman, January 2001, Retrieved on 12-02-2019</ref>. This example is just one of the many examples where the laws come much too late after the technology has been fully developed. <br />
<br />
The same problem is currently happening to drone regulations. Over the last decade, the technological advancements in drones have been enormous, and as a consequence, the accessibility of drones for normal people has increased as well. Nowadays, anyone can buy a drone without any license and fly the drone with a camera to any house in his or her neighbourhood for under 100€ <ref name = "Mediamarkt drone">[https://www.mediamarkt.nl/nl/product/_dji-ryze-tello-powered-by-dji-1556528.html "MediaMarkt Drone: DJI Ryze Tello Powered by DJI", Retrieved on 12-02-2019. </ref>. This seems like an obvious illegal intrusion of privacy by laws such as personality rights ("portretrecht"). However, these rules are not properly enforced concerning drones. In Europe, new drone regulations will be enforced, starting halfway through the year <ref name = "New wet">[https://www.bright.nl/nieuws/artikel/4499011/drones-regels-europa-easa-amsterdam-drone-week "Bright: Nieuwe regels voor drones gaan medio 2019 in"] November 2018, Retrieved on 12-02-2019. </ref>. However, there have been huge debates about how the regulations should be changed, with no concrete answers. Just recently, on January 21 2019, the Dutch House of Representatives ("Tweede Kamer") organised a "rondetafelgesprek", where experts discussed what should be done in terms of regulations<ref name = "rondetafel">[https://www.tweedekamer.nl/debat_en_vergadering/commissievergaderingen/details?id=2018A05009 "Tweede Kamer der Staten-Generaal: Rondetafelgesprek over Drones en killer robots"], January 2019, Retrieved on 13-02-2019 </ref>. These examples show that the regulations of drones are not up to date with the current technological advances of drones.<br />
<br />
=== Limitation of current solutions ===<br />
As we have described before, current solutions such as the eagle experiment, are simply not good enough to efficiently provide a solution to the problem. For this exact reason, airports and governments all over the world are investing vast amounts of money in the development of technologies to counter drones. Heathrow and Gatwick airport are two examples of airports that are investing millions of dollars in this technology <ref name = "invest">[https://www.theguardian.com/world/2019/jan/03/heathrow-and-gatwick-millions-anti-drone-technology "The Guardian: Heathrow and Gatwick invest millions in anti-drone technology", January 2019, Retrieved on 13-02-2019 </ref>.<br />
<br />
Apart from the fact that some solutions simply do not work, other proposed solutions have negative side results. For example, shutting the unwanted UAVs down with radiowaves means that they will crash straight down to the ground. If such a drone falls on someone's head, he or she could get seriously injured. Furthermore, the crashing drone can also break specific equipment when falling. Lastly, if the drone, e.g. falls and breaks on the runway, this could also be dangerous. These consequences also apply to the current solution where the drones are shot down with a laser for example.<br />
<br />
Other solutions such as geofencing and identification signals also have the flaw that they can be bypassed easily. If someone intentionally wants to fly a drone to the airport, it is not that difficult to make sure that the drone does not broadcast identification signals anymore. The drone operator could also make sure that the drone does not send signals that the geofencing uses, such that the geofence is, in fact, useless for deterring this drone. Furthermore, someone could also build a drone themselves, and choose not to send these required signals. This would indeed be against the law in the near future, but since the drone operator is already engaged in criminal activities, these regulations would most likely not stop him. Thus, the technologies can easily be bypassed, rendering them as useless.<br />
----<br />
Back to the [[PRE2018_3_Group4 | root page]].<br />
<br />
= References =<br />
<references /></div>M.g.d.quincey@student.tue.nl